Hyperscalers with Elastic Stack Hyperscalers and Elastic have partnered to enable Managed Service Providers and enterprise IT with a total solution to Security Information Events Management SIEM and security analytics t

Hyperscalers with Elastic Stack

Hyperscalers and Elastic have partnered to enable Managed Service Providers and enterprise IT with a total solution to Security Information Events Management SIEM and security analytics that works out of the box. We have a unique vision for open and modern SIEM that is fast, scalable, and unified. The results – service providers gain the capabilities to outpace adversaries, operate at scale and act decisively. 

The Elastic Stack, often known as the ELK Stack, is utilized in a wide range of use cases, including debugging faults in application metrics, looking into security concerns in logs, and powering search boxes on websites and apps. The Elastic Stack, which consists of Elasticsearch, Kibana, Beats, and Logstash, offers a flexible and adaptable platform for search and analysis for various types of data.

This can be very useful for companies working with large datasets, any complex search application requirements along with infrastructure metrics and container monitoring, logging and application performance monitoring, visualization and analysis of geographical data as well as business and security analytics.

The Elastic stack appliance by Hyperscalers is a complete package including the high-performance CPU, memory and network resources coupled with ELK stack core products to provide corporate search, observability, and security solutions that can be deployed anywhere and are based on a single, adaptable technology stack. 

In the reference guide (Download the Reference Guide in the Downloads Tab), we are trying to provide security to the company data by monitoring their logs, application and analyzing their geographical data. Elasticsearch allows to store, search and analyze huge volumes of data quickly and in near real time and give back answers in milliseconds. It gives quick search responses by searching an index. So highly recommended for those companies, who are looking for data protection of their companies.

Features of Elasticsearch

The Elastic Stack comes with numerous tools (some originally bundled as X-Pack) to let you ingest, analyze, search, and display all forms of data at scale. These features range from enterprise-grade security and developer-friendly APIs to machine learning and graph analytics. The core security features Elastic stack integrates all the security features to the platform are listed below.

a) Authentication: sign on, securely.

b) Authorization: manage users and roles.

c) Encryption: prevent snooping, tampering, and sniffing.

d) Layered security: secure all the way down to the field level.

e) Audit logging: record which user activity.

f) Compliance: meeting security standards

g) Alerting: Highly available, Scalable alerting

h) Monitoring: monitor all types of devices, collect data 

i) Scalability and Resiliency: Elasticsearch operates in a distributed environment designed from the ground up for perpetual peace of mind. Clusters grow with your needs — just add another node.

j) Management: The Elastic Stack comes with a variety of management tools, UIs, and APIs to allow full control over data, users, cluster operations, and more.

k) Alerting: The alerting features of the elastic stack give you the full power of the Elasticsearch query language to identify changes in your data that are interesting to you. In other words, if you can query something in Elasticsearch, you can alert on it.

l) Stack Security: The security features of the Elastic Stack give the right access to the right people. IT, operations, and application teams rely on these features to manage well-intentioned users and keep malicious actors at bay, while executives and customers can rest easy knowing data stored in the Elastic Stack is safe and secure.

m) Deployment: Public cloud, private cloud, or somewhere in between — we make it easy for you to run and manage the Elastic Stack.

n) Clients: The Elastic Stack allows you to work with data in whatever way you're most comfortable. With its RESTful APIs, language clients, robust DSL, and more (even SQL), we're flexible so you don't get stuck.

Infrastructure Setup

To demonstrate a scalable and resilient Elastic Stack, we have used 3 nodes all as master and data. 

Hardware Requirements

The hardware configuration for the build is listed below:

Software Requirements:

Application: Elastic Stack (8.6) 

Product used: Kibana, Logstash, Beats, APM and Elasticsearch Hadoop 

Data visualization: Events, Flows

Ticketing system: Cases

Building Blocks

S5Z | T43Z-2U

The ELK1 node is S5Z | T43Z-2U with high performance multi node server and has high density server optimized for extreme compute performance and space efficiency. 

Featuring all-NVMe with high memory footprint and additional expansibility. Supports top-bin 3rd Generation Intel® Xeon® Scalable processors in compact chassis. Two (2) CPU Sockets for up to 80 cores using Intel® Xeon® Silver 4316 Processor 40cores each. 62.3 Gib of Memory slots of 32 GB each. OS type 64 bit with 16 Front Storage drive bays, 4 for each node. 

Important Considerations

The Elasticsearch architecture is built for scalability and flexibility. The core components are Elasticsearch clusters, nodes, shards, and analyzers. Hyperscalers recommends the below important considerations before proceeding to the deployment phase. 

a. ElasticStack required minimum three nodes to deployed, on each one will be master, and one be data. These nodes can be installed in both bare metal and can be virtualized and on the top of it, we install Elastic Stack products like Kibana, logstash, beats and Elastic Search. 

b. Keep the hardware configuration consistent across all the nodes to ensure replication and high availability.

c. The Elastic stack requires minimum of at least 1 installation of Kibana applications for data visualization and exploration tool for log and time series analytics, applications monitoring and operational intelligence. 

d. Elastic search required minimum of 2 or more cores intel processor with 32 GB memory and 3 hard disks on each node. It supports Linux and MacOS, windows, Debian, Ubuntu and is suitable for Red Hat, Centos SLES, OpenSuSE and other RPM based systems.  

BASE PRODUCT DEPLOYMENT 

Elasticsearch directly deploy on machines in their local data center, it is increasingly common to deploy Elasticsearch in the public cloud or using container orchestrators. We can deploy Elasticsearch on the Amazon and Azure public clouds and via Kubernetes. 

Elastic Cloud on Kubernetes (ECK) supports the deployment of the ELK stack on Kubernetes (including Elasticsearch, Logstash, Kibana and Beats). ECK takes advantage of Kubernetes orchestration capabilities.

ECK allows you to streamline critical operations, including managing and scaling clusters and storage, monitoring multiple clusters, securing clusters and using rolling upgrades for safe configuration. To distribute Elasticsearch resources across availability zones in the cloud, you can enable zone awareness.

You can also set up hot-warm-cold architectures for data storage. ECK lets you tier your data to meet different needs and conserve costs. Hot data is frequently accessed, warm data is infrequently accessed, and cold data is archival or backup storage—you can use lower-cost archive cloud storage tiers for warm and cold data.

Audience and Purpose

Engineers, Enthusiasts, Executives, and IT professionals with background in Computer Science/ Electronics/ Information Technology with understanding in in Linux commands, Java language and basic electronics who intend to study, explore, deploy ELK Stack can be benefitted from this reference guide.

The purpose of this documentation is to provide in depth knowledge about the basic overview, appliance requirements and steps to deploy into your network. 

Documents, Knowledge Base, and Technical Support

Hyperscalers reference architectures and appliance / solutions demonstrations are available at: https://www.hyperscalers.com/OCP-hyperscale-rack-solutions 

For technical queries regarding this document and for managing virtualized, mobile, and cloud technologies, you can contact Hyperscalers technical support at support@hyperscalers.com.

Additional reference to the Elasticsearch, Kibana and security dashboard can be found in Hyperscalers lab as a service (LaaS) page and reference architecture section link – https://elastic.hyperscale2.com

Readers are recommended to have a prior knowledge and expertise with Kibana, Logstash, beats, Linux programming to better understand the following documentation. 

Contact info@hyperscalers.com for more information.

Download the full reference guide in the downloads tab.