tag:blogger.com,1999:blog-81828510797386302422024-03-12T20:23:48.353-07:00EC2Dream - Build and Manage Cloud ServersNeill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-8182851079738630242.post-8721654993273133512020-10-03T08:18:00.058-07:002022-03-21T03:44:00.017-07:00Articles on Medium<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv9Y7wBHCUJGGIBgKMPTGjWro_W6rZ3JRmmkWPGImevlTi6iM-_U7Da0cnHa56nCwgnGzQzZyfxhTsSOxIFB9jGE4NUeINmsjizAa0lVPG_kpdD9rAKiSyZfZoMkW4PB9v0st78DWeY5Y/s2048/pexels-vlada-karpovich-4050347.jpg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="320" data-original-height="1365" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv9Y7wBHCUJGGIBgKMPTGjWro_W6rZ3JRmmkWPGImevlTi6iM-_U7Da0cnHa56nCwgnGzQzZyfxhTsSOxIFB9jGE4NUeINmsjizAa0lVPG_kpdD9rAKiSyZfZoMkW4PB9v0st78DWeY5Y/s320/pexels-vlada-karpovich-4050347.jpg"/></a></div>
<dl>
<dt>I have articles on <a href="https://neillwturner.medium.com/">medium</a>. The following articles can be found there:<br><br></dt>
<dt><a href="https://neillwturner.medium.com/cross-account-aws-managed-prometheus-b56631c74f55">Cross Account AWS Managed Prometheus</a><dt>
<dt><a href="https://neillwturner.medium.com/terraform-and-github-actions-5d0959306674">Terraform and GitHub Actions</a></dt
<dt><a href="https://neillwturner.medium.com/github-actions-self-hosted-runners-on-kubernetes-2cc0da04d41a">Github Actions Self Hosted Runners on Kubernetes</a></dt>
<dt><a href="https://neillwturner.medium.com/kubernetes-for-normal-people-49facc6b0424">Kubernetes for Normal People</a></dt>
<dt><a href="https://neillwturner.medium.com/using-terraformer-to-generate-terraform-files-from-your-existing-cloud-environment-f2923a77d586">Using terraformer to generate terraform files from your existing cloud environment</a></dt>
<dt><a href="https://neillwturner.medium.com/setup-aws-centralised-logs-with-terraform-a2244c807c42">Setup AWS Centralised Logs with Terraform</a></dt>
</dl>
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-26110679020263335612019-06-17T01:01:00.016-07:002020-12-29T08:01:56.816-08:00AWS Techniques<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8su3tbrXI0Rtb011AnockeyawGyZMZbQ3PAbX1VWdX9glLAFV83Lo3WtGmgO8kUU_kIkrck5bOGxEEUKF5cewtjLl-98fSrJt30IBsUrVw3EC-7CC49joGvP3Zfl_U7ru0rhK_Rjr9yg/s1600/AWS.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8su3tbrXI0Rtb011AnockeyawGyZMZbQ3PAbX1VWdX9glLAFV83Lo3WtGmgO8kUU_kIkrck5bOGxEEUKF5cewtjLl-98fSrJt30IBsUrVw3EC-7CC49joGvP3Zfl_U7ru0rhK_Rjr9yg/s320/AWS.png" width="320" height="213" data-original-width="275" data-original-height="183" /></a></div>
<dl>
<dt><b>VPC</b></dt>
<dd>- use the terraform <a href="https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest">VPC module</a> to abstract away the complexity of network resources.</dd>
<dt><br><b>AWS Firewall options</b></dt>
<dd>chose your firewall options as per <a href="https://blog.doit-intl.com/aws-firewalls-101-how-and-when-to-use-each-one-d4ad8087a6b3">AWS Firewalls 101</a></dd>
<dt><br><b>Understanding IAM policies</b></dt>
<dd><a href="https://medium.com/tensult/aws-policies-with-examples-8340661d35e9">AWS IAM Policies with Examples</a></dd>
<dd><a href="https://jayendrapatil.com/tag/iam/">IAM</a></dd>
<dd><a href="https://start.jcolemorrison.com/aws-iam-policies-in-a-nutshell/">AWS IAM Policies in a Nutshell</a></dd>
<dd><a href="https://sonraisecurity.com/blog/aws-iam-common-mistakes/">AWS IAM Breakdown… and common mistakes!</a></dd>
<dd><a href="https://www.e4developer.com/2019/07/28/aws-iam-looking-at-some-of-the-more-advanced-features/">AWS IAM – Looking at some of the more advanced features</a></dd>
<dd><a href="https://gruntwork.io/guides/foundations/how-to-configure-production-grade-aws-account-structure/">How to configure a production-grade AWS account structure</a></dd>
<dd><a href="https://scalesec.com/aws-series/5-things-you-should-be-doing-in-iam-right-now/">5 Things You Should Be Doing in IAM Right Now</a></dd>
<dd><a href="https://blog.cloudsploit.com/privilege-escalation-in-amazon-web-services-cb4837365958#.ob8807stq">Elevating Permissions in AWS IAM</a></dd>
<dd><a href="https://nodramadevops.com/2019/11/why-is-aws-iam-so-hard/">Why is AWS IAM So Hard?</a></dd>
<dd><a href="https://www.serverless.com/blog/abcs-of-iam-permissions">The ABCs of IAM: Managing permissions with Serverless</a></dd>
<dt><br>also don't forget Service control policies in AWS Organisations</dt>
<dd><a href="https://blog.scalesec.com/using-terraform-to-secure-your-aws-organizations-399c3dcb4b5a">Using Terraform to Secure Your AWS Organizations</a></dd>
<dd><a href="https://medium.com/slalom-engineering/aws-multi-account-architecture-with-terraform-yeoman-and-jenkins-7fd42ddcdda8">AWS Multi-Account Architecture with Terraform, Yeoman, and Jenkins</a></dd>
<dd><a href="https://www.reddit.com/r/aws/comments/b58nv8/aws_organizations_pro_tips/">AWS Organizations pro tips</a></dd>
<dt><br><b>Process to write, test policies and deploy IAM Policies.</b></dt>
<dt>see <a href="https://www.1strategy.com/blog/2018/06/19/tips-for-creating-iam-policies/">Tips for Creating IAM Policies</a></dt>
<dd><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html">IAM Policy Reference</a></dd>
<dd><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html">AWS Services That Work With IAM</a></dd>
<dd><a href="https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html">AWS ARNs and AWS Service Namespaces</a></dd>
<dt>1. Google for “Limiting Access to AWS X Service” and then modify the policy.</dt>
<dt>2. Generate Policy</dt>
<dd><a href="https://awspolicygen.s3.amazonaws.com/policygen.html">AWS Policy Generator</a></dd>
<dt>3. Test Policy</dt>
<dd><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html">Testing IAM policies with the IAM policy simulator</a></dd>
<dt>4. S3 Bucket Policies</dt>
<dd>Add deny section to deny any iam roles</dd>
<dd><a href="https://nodramadevops.com/2020/04/why-protecting-data-in-s3-is-hard-and-a-least-privilege-bucket-policy-to-help/">Why protecting data in S3 is hard and a least-privilege bucket policy to help</a></dd>
<dt>5. Deploy with terraform</dt>
<dd>-run <a href="https://dev.to/securitylater/securing-terraform-code-with-tfsec-3980">tfsec</a> to do static analysis</dd>
<dd>-terraform apply</dd>
<dd>-test its working.</dd>
<dt><br><b>Tools</b></dt>
<dd>security tools to help write and test IAM policies</dd>
<dd>security tools to monitor etc security i.e. security operations.</dd>
<dt>AWS Console - Visual Policy Editor</dt>
<dd><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html#access_policies_create-start">Creating IAM policies</a></dd>
<dt>AWS Policy Generator (static website)</dt>
<dd><a href="https://awspolicygen.s3.amazonaws.com/policygen.html">AWS Policy Generator</a></dd>
<dt>IAM policy simulator</dt>
<dd><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html">Testing IAM policies with the IAM policy simulator</a></dd>
<dd><a href="https://www.qualimente.com/2019/03/11/testing-an-s3-policy-using-the-aws-iam-simulator/">Testing an S3 policy using the AWS IAM Simulator</a></dd>
<dt>iamctl</dt>
<dd><a href="https://aws.amazon.com/blogs/security/new-iamctl-tool-compares-multiple-iam-roles-and-policies/">New IAMCTL tool compares multiple IAM roles and policie</a></dd>
<dt>aws-iam-policy-tool</dt>
<dd><a href="https://github.com/tilfin/aws-iam-policy-tool">tilfin/aws-iam-policy-tool</a></dd>
<dd><a href="https://snyk.io/advisor/npm-package/aws-iam-policy-tool">aws-iam-policy-tool</a></dd>
<dt>policy sentry</dt>
<dd><a href="https://policy-sentry.readthedocs.io/en/v0.11.0/index.html">Policy Sentry Documentation</a></dd>
<dd><a href="https://policy-sentry.readthedocs.io/en/v0.11.0/tutorial/">Policy Sentry Tutorial</a></dd>
<dt>automated security testing.</dt>
<dd><a href="https://pypi.org/project/aws-iam-tester/">aws-iam-tester</a></dd>
<dt>AWSume</dt>
<dd><a href="https://awsu.me/">AWSume: AWS Assume Made Awesome!</a></dd>
</dl>
<dt><br>Security</dt>
<dt><a href="https://blog.doit-intl.com/aws-firewalls-101-how-and-when-to-use-each-one-d4ad8087a6b3">AWS Firewalls 101</a><br /></dt>
<dt><a href="https://www.ncsc.gov.uk/whitepaper/security-architecture-anti-patterns">Security architecture anti-patterns</a><br /></dt>
<dt><a href="https://medium.com/swlh/how-to-integrate-security-on-the-devops-pipeline-e36dea836d7b">How can we integrate security into the DevOps pipelines?</a><br /></dt>
<dt><a href="https://www.xplg.com/s3-security-buckets/">AWS S3 Security Guide</a><br /></dt>
<dt><a href="https://slack.engineering/moving-fast-and-securing-things-540e6c5ae58a">Moving Fast and Securing Things - The SDL at Slack and goSDL</a><br /></dt>
<dt><a href="https://medium.com/stax-blog/a-guide-to-tagging-resources-in-aws-8f4311afeb46">A guide to tagging resources in AWS</a><br /></dt>
<dt><a href="https://medium.com/@Markus.Hanslik/setting-up-an-ssl-certificate-using-aws-and-terraform-198c6fb90743">hsetting up an ssl certificate using aws and terraform</a><br /></dt>
<dt><a href="https://blog.newrelic.com/technology/critical-devops-interview-questions-and-how-to-answer-them">critical devops interview questions and how to answer them</a><br /></dt>
<dt><a href="https://www.slideshare.net/JasonHarley3/a-year-in-production-with-the-hashistack">A year in Production with the Hashistack</a><br />
</dt>
<dt><a href="https://serverlesspatterns.io/">Serverless Patterns</a><br />
<dt><a href="https://www.rackspace.com/en-gb/blog/creating-cloud-governance-in-an-agile-world">Creating cloud governance in an agile world</a><br />
</dt>
<dt><br>Other</dt>
<dt><a href="https://start.jcolemorrison.com/the-hitchhikers-guide-to-aws-ecs-and-docker/">The Hitchhiker's Guide to AWS ECS and Docker</a><br /></dt>
<dt><a href="https://github.com/christopherhein/terraform-eks">Terraform EKS Cluster Operations - github sample</a><br /></dt>
<dt><a href="https://learn.hashicorp.com/terraform/aws/eks-intro">Terraform AWS EKS Introduction</a></dt>
<dt><a href="https://medium.com/better-programming/a-whole-year-of-amazon-eks-805e13d9600c">A Whole Year of Amazon EKS</a></dt>
<dt><a href="https://medium.com/@Joachim8675309/deploying-helm-charts-w-terraform-58bd3a690e55">Deploying Helm Charts w. Terraform</a></dt>
<dt><a href="https://medium.com/better-programming/troubleshooting-kubernetes-the-beginners-guide-2440ec400155">Troubleshooting Kubernetes: The Beginner's Guide</a><br /></dt>
<dt><a href="https://start.jcolemorrison.com/aws-vpc-core-concepts-analogy-guide/">AWS VPC Core Concepts in an Analogy and Guide</a><br /></dt>
<dt><a href="https://www.rundeck.com/blog/whats-a-silo-and-why-they-ruin-everything">What's a Silo? (and why they ruin everything)</a><br /></dt>
<dt><a href="https://www.rundeck.com/self-service">Self-Service Operations</a><br /></dt>
</dl>Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-92176202461958772182018-11-19T07:18:00.000-08:002020-02-28T05:44:44.075-08:00DevSecOps<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdLFtWcv2HwRds-yzCHyPqaJh99fwusnJ192MPyF_7R5vdcSmjSeU1_J2_-ikYdeVDOpGffZKg5WHIGlEIHtUPCKk9acPup2scSB9ZHYvvXCRbelxPdHDyA48BwgRqgU1GMx5bLjOpDQk/s1600/security.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdLFtWcv2HwRds-yzCHyPqaJh99fwusnJ192MPyF_7R5vdcSmjSeU1_J2_-ikYdeVDOpGffZKg5WHIGlEIHtUPCKk9acPup2scSB9ZHYvvXCRbelxPdHDyA48BwgRqgU1GMx5bLjOpDQk/s200/security.jpeg" width="200" height="111" data-original-width="302" data-original-height="167" /></a></div>
<b>DevSecOps</b> is really about
<ul>
<li>automation - getting security into CI/CD pipeline.</li>
<li>monitoring -
<ul>
<li>tracking compliance across the multiple cloud accounts and providers.</li>
<li>metrics to better track our cloud security posture</li>
</li>
</ul>
</ul>
<b>Cloud Security Checklist</b>
<ul>
<li><b>Directory service.</b></br> If you use identity and access management, you need a directory to keep the identities. Although Microsoft’s Active Directory works just fine, any LDAP-compliant directory will work. Note that you need to deal with security at the directory level as well, so the directory itself does not become a vulnerability.</li>
<li><b>Identity and access management.</b></br> IAM is needed to ensure that you can configure who is who, who is authenticated, and what devices, applications, or data they can access. This gives you complete control over who can do what, and it puts limits on what they can do. These IAM tools are either native to the public cloud platform or come from a third party.</li>
<li><b>Encryption services.</b></br> What specific encryption you needwill largely depend on where you are in the world and the types of things you need to encrypt, as well as if you need to encrypt data at rest, in flight, or both. I say “services” (plural) because you’ll likely ise more than one encryption service, including at the file, database, and network levels.</li>
<li><b>Security ops.</b></br> Often overlooked, this is the operational aspect of all of security. Security ops, aka secops, includes the ability to proactively monitor the security systems and subsystems to ensure that they are doing their jobs and that the security services are updated with the latest information they need to keep your system safe.</li>
<li><b>Compliance management.</b></br> Another often overlooked security feature, this is where you deal with those pesky rules and regulations that affect security. No matter if you need to be GDPR-compliant or HIPAA-compliant, this is where you have a console that alerts you to things that may be out of compliance and lets you take corrective action.</li>
</ul>
<b>References</b></br>
<a href="https://thenewstack.io/how-devops-can-save-security/">How DevOps Can Save Security</a></br>
<a href="https://www.infoworld.com/article/3321757/cloud-security/cloud-security-the-essential-checklist.html"> Infoworld - Cloud security: The essential checklist</a></br>
<a href="https://www.skyhighnetworks.com/cloud-security-blog/13-aws-iam-best-practices-for-security-and-compliance/">13 AWS IAM Best Practices for Security and Compliance</a></br>
<a href="https://medium.com/@marioplatt/whats-the-fuss-with-compliance-as-code-11ac8e985dc">whats the fuss with compliance as code</a></br>
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-39614642321738171872017-01-05T09:07:00.000-08:002017-09-09T11:52:02.128-07:00Verify AWS Infrastructure with Test Kitchen and awspec<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMoKEiM7vsPCkC0goVnH7PwkIsLNsWAjV9Du27qEYjCNBSZMi8CF5L-ZmDcCcwyzq_hkVpWxUZOYCv71c4zST8df6hRHXpKCRRpoF9CkyMZJXElGLrL-dw5i22lFykGqIKOpml3WJQMyc/s1600/256px-Cloud_computing_icon.svg.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMoKEiM7vsPCkC0goVnH7PwkIsLNsWAjV9Du27qEYjCNBSZMi8CF5L-ZmDcCcwyzq_hkVpWxUZOYCv71c4zST8df6hRHXpKCRRpoF9CkyMZJXElGLrL-dw5i22lFykGqIKOpml3WJQMyc/s200/256px-Cloud_computing_icon.svg.png" width="200" height="140" /></a></div><a href="http://kitchenci.com">Test-kitchen</a> is a test harness tool that allows integration testing of configuration and infrastructure.<br><br>
It has a plug-in architecture and has plugins for both <a href="https://aws.amazon.com/cloudformation/">Cloud Formation</a> and <a href="https://www.terraform.io/">terraform</a> to create AWS infrastructure. For plugin details see <a href="https://github.com/newcontext-oss/kitchen-terraform">kitchen-terraform</a> and <a href="https://github.com/neillturner/kitchen-cloudformation">kitchen-cloudformation</a>.<br><br>
<a href="https://github.com/k1LoW/awspec">awspec</a> is a tool that allows RSpec tests for AWS resources. For plugin details see
<a href="https://github.com/neillturner/kitchen-verifier-awspec">kitchen-awspec</a>.<br><br>
There is a simple example in the kitchen-awspec github repository to demonstrate creating an EC2 instance with cloud formation and then verifying it with awspec. See <a href="https://github.com/neillturner/kitchen-verifier-awspec/tree/master/example">https://github.com/neillturner/kitchen-verifier-awspec/tree/master/example</a><br><br>
1. <b>Install test-kitchen as per <a href="http://kitchen.ci/docs/getting-started/installing">http://kitchen.ci/docs/getting-started/installing</a></b><br>
basically install ruby for your platform <br>
then to install test-kitchen<br>
gem install test-kitchen<br><br>
and the cloud formation or terraform plugin:<br>
gem install kitchen-cloudformation or gem install kitchen-terraform<br><br>
and the awspec plugin<br>
gem install kitchen-verifier-awspec<br><br>
2. <b>Use git to clone the github kitchen-awspec repository.</b><br>
git clone https://github.com/neillturner/kitchen-verifier-awspec.git<br> <br>
3. <b>Run the example on your workstation:</b><br><br>
a. Configuration AWS Settings<br>
See <a href="http://docs.aws.amazon.com/ja_jp/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence">Configuration Settings and Precedence</a><br>
For example:<br>
# configure aws keys via either a profile<br>
export AWS_DEFAULT_PROFILE=myprofile<br>
# or set keys directly<br>
export AWS_ACCESS_KEY_ID=myaccesskey<br>
export AWS_SECRET_ACCESS_KEY=mysecretkey<br><br>
# set aws region<br>
export AWS_REGION=eu-west-1<br><br>
b. Go to example directory<br>
cd kitchen-verifier-awspec/example<br><br>
c. For windows set the ssl cert file<br>
set SSL_CERT_FILE=C:/repository/kitchen-verifier-awspec/example/ca-bundle.crt<br><br>
d. Check that everything installed successfully<br>
kitchen list<br>
should return the base-aws instance<br><br>
e. Create the Cloud Formation stack<br>
kitchen create base-aws -l debug<br><br>
f. Verify the AWS Resources<br>
kitchen verify base-aws -l debug<br><br>
g. Delete the Cloud Formation stack<br>
kitchen destroy base-aws -l debug<br><br>
<b>References</b><br>
<a href="http://www.slideshare.net/stelligent/testdriven-infrastructure">http://www.slideshare.net/stelligent/testdriven-infrastructure</a><br>
<a href="http://www.slideshare.net/dkcwd/validating-puppet-managed-resources-in-aws-with-awspec">http://www.slideshare.net/dkcwd/validating-puppet-managed-resources-in-aws-with-awspec</a><br>
<a href="https://www.newcontext.com/introducing-kitchen-terraform/">https://www.newcontext.com/introducing-kitchen-terraform/</a><br>
<a href="https://www.unixdaemon.net/cloud/testing-terraform-projects/">https://www.unixdaemon.net/cloud/testing-terraform-projects/</a><br>
<a href="http://koesystems.github.io/terraform-best-practices/">http://koesystems.github.io/terraform-best-practices/</a><br>
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-69555417087307496362015-12-27T06:12:00.000-08:002016-12-05T05:55:31.132-08:00Automated Devops with Test-Kitchen, Ansible and AnsibleSpec<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTD7Y7uB01ZED_jyAFFzJpKvGR4ogNESzC6C4ooIN7RnQNkGrZTmhoNA0vN2h0d8lTyqEVdtO5ivlOvbKw8DxLJ_dQQpZAhJCTiCCi4IcRCqkzDL9In9VdB_43Aee9FEVIlDjMMJLGeuE/s1600/ansiblespec.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTD7Y7uB01ZED_jyAFFzJpKvGR4ogNESzC6C4ooIN7RnQNkGrZTmhoNA0vN2h0d8lTyqEVdtO5ivlOvbKw8DxLJ_dQQpZAhJCTiCCi4IcRCqkzDL9In9VdB_43Aee9FEVIlDjMMJLGeuE/s200/ansiblespec.jpg" /></a></div>
This demonstrates using test-kitchen, ansible and ansiblespec to build and verify an apache server.</br></br>
Ansible and apache can be installed and configured on the same server.</br></br> Alternatively ansible and apache can be installed and configured on separate servers. In this case:</br>
-Everything is done via ssh from the Ansible/Serverspec server so nothing is installed on the apache server.</br>
-In this demonstration both servers are centos 7 running like Amazon EC2, or a Docker Container as long as they are accessible via ssh.</br>
-You can take an image of the server after it is build and no comfiguration software is install on the apache Server.</br>
-this is using ansible in ssh connection mode to do remote configuration.</br></br></br>
<div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVD6rabBgAcBI-FSxqGWK2amee8tqfAmJsgMynQWHmniBEGsDkCHyidaqPz5PMvdQeDEp29RFcTsYR3hfwKt52UkKAlrVq3ZLEJbbGQ1wvg4IJ0gXZrkKZUxXjKo2ocu862F8BdA8h5hs/s1600/kitchen-ansible.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVD6rabBgAcBI-FSxqGWK2amee8tqfAmJsgMynQWHmniBEGsDkCHyidaqPz5PMvdQeDEp29RFcTsYR3hfwKt52UkKAlrVq3ZLEJbbGQ1wvg4IJ0gXZrkKZUxXjKo2ocu862F8BdA8h5hs/s400/kitchen-ansible.png" width="400" height="225" /></a></div>
</div>
</br>
<h2>Workstation Software Installation</h2>
The first thing you need to do is install the test-kitchen environment on your workstation.</br>
A useful link is:
<a href='http://misheska.com/blog/2013/12/26/set-up-a-sane-ruby-cookbook-authoring-environment-for-chef'>http://misheska.com/blog/2013/12/26/set-up-a-sane-ruby-cookbook-authoring-environment-for-chef</a></br>
The follow instructions are for Windows PC (it will be similar for Mac):</br></br>
1. Download and install the Windows RubyInstaller for 32 bit Ruby 2.1 from <a href='http://rubyinstaller.org/downloads'>http://rubyinstaller.org/downloads</a>.</br>
Check the option to add ruby to your path.</br></br>
2. Download and install the Windows Ruby DevKit for use with Ruby 2.0 and above (32bits version only) from <a href='http://rubyinstaller.org/downloads'>http://rubyinstaller.org/downloads</a>.</br></br>
3. Configure the Ruby DevKit</br>
In the devkit directory run “ruby dk.rb init”.</br>
Check the config.yml generated has added the the path of the ruby install, if not add it manually.</br>
run “ruby dk.rb install” to bind it to the ruby installation.</br></br>
4. Then install the following gems</br>
gem install test-kitchen</br>
gem install kitchen-ansible</br>
gem install kitchen-verifier-serverspec</br></br>
5. Install a git client on your workstation and then git clone the repository
<a href='https://github.com/neillturner/ansible_ansiblespec_repo'>https://github.com/neillturner/ansible_ansiblespec_repo</a></br> and in a command window in the ansible_repo directory run command
</br>
kitchen list
</br>
This will return a list if everything is correctly installed.
</br></br>
<h2> ansible and apache on the same server.</h2>
1. Create 1 linux server for both ansible and apache using a keypair using say AWS Cloud Formation.</br></br>
2. In ansible_ansiblespec_repo update the inventory/hosts_ssh with IP address of linux server.</br></br>
3. In the .kitchen.yml file</br>
* Set the ssh_key to the aws keypair for linux server e.g. spec/test.pem</br>
* Set the hostname to ip address of linux server e.g.'54.229.103.38'</br></br>
4. create, converge, verify and destroy the ansible-centos-70 server</br></br>
kitchen create ansible-centos-70 -l debug</br>
kitchen converge ansible-centos-70 -l debug</br>
kitchen verify ansible-centos-70 -l debug</br>
kitchen destroy ansible-centos-70 -l debug</br>
<h2>ansible and apache on separate servers.</h2>
1. Create 2 linux servers one for ansible and one for apache using a keypair using say AWS Cloud Formation.</br></br>
2. In ansible_ansiblespec_repo update the inventory/hosts_ssh with IP address of apache server.</br></br>
3. In the .kitchen.yml file</br>
* Set the ssh_key to the aws keypair for ansible and apache server e.g. spec/test.pem</br>
* Set the hostname to ip address of ansible server e.g.'54.229.103.38'</br></br>
4. create, converge, verify and destroy the ansible-centos-70 server</br></br>
kitchen create ansible-centos-70 -l debug</br>
kitchen converge ansible-centos-70 -l debug</br>
kitchen verify ansible-centos-70 -l debug</br>
kitchen destroy ansible-centos-70 -l debug</br>
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-60461071689726635022015-06-20T05:22:00.000-07:002018-12-05T08:58:22.464-08:00Immutable Infrastructure<blockquote><i>"One of my mantras - focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains."</i> - Steve Jobs</blockquote><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimPQWsjw8dZB89rWIwOSLPPg60Wg-oNcLI_3Ae-LxVL4GQNHAcRHpL49DPii4t0-6SmY1tXp6ZHew18hDfrfJnfSkxDbj6LA0PAqaiD_dKxmMEyzqnK58cyl01yzBOhLi7IDYtt-82suU/s1600/immutable.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimPQWsjw8dZB89rWIwOSLPPg60Wg-oNcLI_3Ae-LxVL4GQNHAcRHpL49DPii4t0-6SmY1tXp6ZHew18hDfrfJnfSkxDbj6LA0PAqaiD_dKxmMEyzqnK58cyl01yzBOhLi7IDYtt-82suU/s320/immutable.png" width="320" height="184" data-original-width="657" data-original-height="378" /></a></div>
<br/><b>Mutable Infrastructure</b> is using Chef Server, Puppet Master or Ansible Tower to pull running updates regularly on all your servers.
<br/><br/>Instead use:
<ul>
<li><b>Immutable infrastructure</b> where ever possible - replace servers instead of updating them.</li><br/>
<li><b>Serverless infrastructure.</b> Cloud services like AWS RDS, Lamba, SQS etc.</li><br/>
<li>Use CI tools like <b>Jenkins</b> to build application artifacts to use in <b>Packer</b> images and <b>Docker</b> containers.</li></br>
<li><b>Custom Images or Containers.</b> Use <a href="https://packer.io/">Packer</a> or <b>Docker</b> in development to build custom images or containers to use in test and live.</li><br/>
<li>Only one <b>Git</b> repository for all environments - use a production branch and allow other branches and allow servers to be built from different branches.</li><br/>
<li><b>Terraform</b> to orchestrate the infrastructure using the packer images or Docker containers.</li><br/>
<li>Don't use Configuration Management tools for deployment of applications.</li><br/>
</ul>
It works best with a cloud:
<ul>
<li>The AWS Console know where your servers are and what their roles are instead of chef server or puppermaster.</li><br/>
<li>Orchestrate the servers with terraform or cloudformation.</li><br/>
<li>Monitor the servers, manage the log files and generate alarms (e.g. AWS CloudWatch).</li><br/>
<li>Run your databases as services (e.g. AWS RDS).</li><br/>
</ul>
Try and keep it simple and build as little as possible yourself as this will make you as efficient as possible.<br/>
<br/><br/><b>References</b><br/>
<a href="https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-puppet-ansible-saltstack-or-cloudformation-7989dad2865c">Why we use Terraform and not Chef, Puppet, Ansible, SaltStack, or CloudFormation</a><br/>
<a href="http://natishalom.typepad.com/nati_shaloms_blog/2015/06/devops-is-not-a-feature.html">DevOps is Not a Feature!</a><br/>
<a href="http://blog.xebialabs.com/2014/05/08/9-reasons-provisioning-tools-arent-ideal-deploying-enterprise-apps-2/">9 Reasons Provisioning Tools Aren’t Ideal for Deploying Enterprise Apps</a><br/>
<a href="http://www.techinsight.io/review/devops-and-automation/testing-infrastructure-with-test-kitchen/">
Testing infrastructure with Test Kitchen</a><br/>
<a href="http://ringo.de-smet.name/2015/03/keep-chef-out-of-your-docker-containers/">Keep Chef out of your Docker containers</a><br/>
<a href="https://www.greenreedtech.com/vsphere-immutable-infrastructure-with-terraform/">vSphere Immutable Infrastructure with Terraform</a><br/>
<a href="https://sgotti.me/post/terraform-immutable-infrastructure-stateful-rolling-upgrades/">Immutable infrastructure with terraform and rolling upgrades of stateful services</a></br>Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-66056138964584101072014-02-20T08:52:00.000-08:002015-09-18T07:47:50.088-07:00Test Driven Development with Chef, Puppet and Ansible<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrDDmO9fh92fFWDtHKt9HxeqnQ6nLbamSMB02uUr578FlZ9Wm6-7KXBdOdr992fau-boWPUACvEX1V51pMZr5BWK5N29VV5VDRw11lg1ss-FVLBs3jhU4MMvSawDIgnkrL4BTElcl0Sak/s1600/test-tubes-1258732-m.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrDDmO9fh92fFWDtHKt9HxeqnQ6nLbamSMB02uUr578FlZ9Wm6-7KXBdOdr992fau-boWPUACvEX1V51pMZr5BWK5N29VV5VDRw11lg1ss-FVLBs3jhU4MMvSawDIgnkrL4BTElcl0Sak/s200/test-tubes-1258732-m.jpg" /></a></div>
<i>"build the right thing and the thing right"</i><br /><br />
An increased focus on testing, continuous integration and agile development is occuring in
software development. This is also influencing configuration tools Chef, Puppet and Ansible.
We are seeing influential books like <a href="http://it-ebooks.info/book/3088/">Test-Driven Infrastructure with Chef, 2nd Edition</a> emerge.<br /><br />
The following tools seem to be becoming well established:<br>
</p><table>
<tbody><tr><td><b>Test Type</b></td><td><b>Chef</b></td><td><b>Puppet</b></td><td><b>Ansible</b></td></tr>
<tr><td><i>Style</i></td><td>Foodcritic</td><td>Puppet-lint</td><td>Ansible-lint</td></tr>
<tr><td><i>Unit</i></td><td>Chefspec</td><td>Rspec-Puppet</td><td>Rspec</td></tr>
<tr><td><i>Integration </i></td><td>Serverspec </td><td>Serverspec or Beaker-rspec</td><td>Serverspec or Ansible_spec</td></tr>
<tr><td><i>Test Harness </i></td><td>Test Kitchen </td><td>Test Kitchen or Beaker</td><td>Test Kitchen</td></tr>
</tbody></table><br>
<b>Style and Syntax Testing</b><br>
Foodcritic, Puppet-lint and Ansible-lint can quickly test the syntax and style of your cookbooks/modules on your own workstation without writing any tests so this is the easiest testing to do.<br>
See explanations of <a href='http://acrmp.github.io/foodcritic/#FC001'>Foodcritic messages</a> or <a href='http://puppet-lint.com/checks/'>Puppet-lint checks</a> or <a href='https://github.com/willthames/ansible-lint'>ansible-lint</a><br><br>
<b>Unit Testing</b><br>
Chefspec and rspec-puppet are good for regression testing and checking that the cookbooks or modules build what they are suppose to be building. Here you have to write the tests in rspec format but you don't need to have a server and run chef/puppet because it simulates this. But there are limitations so not all features of your cookbook/module can be testing.<br>
To learn chefspec see <a href="https://github.com/jimhopp/chefspec_exploration">chefspec exploration</a>, <a href="https://github.com/sethvargo/chefspec/tree/master/examples">chefspec examples</a>, <a href="http://code.sethvargo.com/chefspec/">chefspec doc</a> and <a href="http://rubydoc.info/github/sethvargo/chefspec">rubydoc</a><br>
To learn rspec-puppet see <a href="http://rspec-puppet.com/tutorial/">rspec-puppet tutorial</a> and the <a href="http://rspec-puppet.com/matchers/">matches</a>
<br><br>
<b>Integration Testing</b><br>
Once again Serverspec tests need to be written but they are in a similar format to rspec tests. However integration tests are
more time consuming as it requires chef or puppet to be run on a remote server to converge it before running the tests.<br> Because chef/puppet/ansible converge the node you don't need to write tests to prove that chef/puppet/ansible works. The tests should be equivalent to what system admin does when they log on to the server and check various files, directory and logs to check all is ok.<br>
To learn serverspec see <a href="http://serverspec.org/resource_types.html">Resource Types</a> and <a href='https://github.com/sho-h/serverspec-examples/tree/master/spec/localhost'>serverspec examples</a><br>
<a href="https://github.com/puppetlabs/beaker/wiki/How-to-Write-a-Beaker-Test-for-a-Module">Beaker-rspec</a> is also in rspec format and see the <a href="https://github.com/puppetlabs/beaker/wiki/The-Beaker-DSL-API">Beaker DSL with examples</a>.
<br><br>
<b>Test Harness</b><br>
<a href="http://kitchen.ci/">Test Kitchen</a> is fast becoming the standard way to run integration tests for chef as it automatically creates the servers and runs the tests and logs the results. It has a great <a href="http://kitchen.ci/docs/getting-started/"> Getting Started Tutorial</a>.<br>
See Test Kitchen <a href="https://github.com/test-kitchen/test-kitchen/wiki/Getting-Started">Getting Started</a> for test kitchen commands.<br>See demo of test-kitchen 1.0 <a href="https://github.com/bryanwb/tk-demo">chef repository</a><br>
<b>Puppet</b> is supported in Test Kitchen using the <a href="https://github.com/neillturner/kitchen-puppet">kitchen-puppet</a> plugin.<br>
<b>Ansible</b> is supported in Test Kitchen using the <a href="https://github.com/neillturner/kitchen-ansible">kitchen-ansible</a> plugin.<br><br>
<a href="https://github.com/puppetlabs/beaker">Beaker</a> is the puppet developed equivalent of Test Kitchen. It supports provisioning of servers in vagrant or AWS etc. Unlike Test Kitchen it has its own beaker-rspec language for writing tests similar to serverspec. Because it is ruby code other testing tools can be called from this code.<br>
See <a href="https://github.com/puppetlabs/beaker/wiki">Beaker wiki</a> for documentation. Currently there are not many examples of creating beaker DSL tests.
<br><br>
<a href="https://github.com/volanja/ansible_spec">Ansible-spec</a> is an alternative to test-kitchen and serverspec. It has a particular directory structure to follow <a href='https://github.com/volanja/ansible-sample-tdd'>ansible-sample-tdd</a>. It should be possible to call ansible-spec from test-kitchen if so desired.
<br><br>
<b>Continuous Integration</b><br>
We are starting to see <a href="http://www.slideshare.net/bamdadd/automated-deployment-pipeline-using-jenkins-puppet-mcollective-and-aws">continuous integration pipelines</a> being set in systems like Jenkins that automate the test process.<br><br>
<b>Auditing</b><br>
Many companies run puppet with the --noop option regularly over the server estate to check the configuration are correct. Ansible also support a similar option called dryrun. However often these option canno fully simulate complex modules or playbooks so they are not a guarantee that they will work with run for real.<br><br>
EC2Dream facilitates both cookbook development and testing development by supplying a sample repository and:<br>
a. A graphical interface to test kitchen<br>
b. The ability to push a repository and tests to a remote machine and run chef or puppet and the test tools.<br><br>
Chef References<br>
<a href="http://acrmp.github.io/foodcritic/">foodcritic docs</a><br>
<a href="http://code.sethvargo.com/chefspec/">chefspec docs</a><br>
<a href="http://www.neverstopbuilding.com/chefspec">Outside In TDD with Chefspec</a><br>
<a href="http://leopard.in.ua/2013/12/01/chef-and-tdd/">Chef cookbooks development by TDD</a><br>
<a href="http://sysadvent.blogspot.co.uk/2013/12/day-11-lazy-sysadmins-guide-to-test.html">Day 11 Lazy Sysadmins Guide to Test Driven Chef Cookbooks</a><br>
<a href="http://dracoater.blogspot.co.uk/2013/09/testing-chef-cookbooks-part-2-chefspec.html">Testing Chef Cookbooks. Part 2. Chefspec</a><br>
<a href="https://sethvargo.com/unit-testing-chef-cookbooks/">Unit Testing Chef Cookbooks</a><br>
<a href="https://sethvargo.com/chef-recipe-code-coverage/">Chef Recipe Code Coverage</a><br>
<a href="https://engineering.aweber.com/test-driven-chef-cookbooks-with-test-kitchen/">Test-Driven Chef Cookbooks with Test Kitchen</a><br>
<a href="http://blog.pagerduty.com/2013/11/chef-testing-pagerduty/">Chef Testing at PagerDuty</a><br>
<a href="http://www.slideshare.net/JulianDunn/cookbook-testing-and-ci-chefboston">Chef Cookbook Testing and Continous Integration</a><br><br>
Puppet References<br>
<a href="http://betterspecs.org/">BetterSpecs</a><br>
<a href="http://rspec-puppet.com/tutorial/">Rspec Puppet Tutorial</a><br>
<a href="http://www.slideshare.net/PuppetLabs/automated-puppet-testing-puppetcamp-chicago-12-scott-nottingham">Automated Puppet Testing</a><br>
<a href="http://puppetlabs.com/blog/verifying-puppet-checking-syntax-and-writing-automated-tests">Verifying Puppet: Checking Syntax and Writing Automated Tests</a><br>
<a href="http://puppetlabs.com/blog/test-driven-development-with-puppet">Test Driven Development with Puppet</a><br>
<a href="http://puppetlabs.com/blog/the-next-generation-of-puppet-module-testing">The Next Generation of Puppet Module Testing</a><br>
<a href="http://genuxation.com/wiki/index.php/RSpec_tests_for_puppet_modules">RSpec tests for puppet modules</a><br>
<a href="http://terrarum.net/development/puppet-testing-part-1.html">Puppet Testing Part 1</a><br>
<a href="http://blog.csanchez.org/tag/rspec/">Testing puppet modules</a><br>
<a href="http://vstone.eu/puppet-modules-in-jenkins/">Puppet modules in Jenkins.</a><br><p></p>Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-16516827147422988712013-10-19T01:32:00.000-07:002014-03-04T08:18:33.443-08:00Puppet Roles and Parameter Hierarchies like Chef <div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxEEGElgcChi6UpAlIUKtF_lEtOjs2BLY7990BScRCyA1UBLiftqMH96JCZcUe1op82u6X4Fk9tCCQumS673kxUY_JFhm2BZE7R5RcAI6E9_HNUd821nbmRr1GRCxSrmLgv4YmeaS9Rfg/s1600/diversity-5-840316-m.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxEEGElgcChi6UpAlIUKtF_lEtOjs2BLY7990BScRCyA1UBLiftqMH96JCZcUe1op82u6X4Fk9tCCQumS673kxUY_JFhm2BZE7R5RcAI6E9_HNUd821nbmRr1GRCxSrmLgv4YmeaS9Rfg/s200/diversity-5-840316-m.jpg" /></a></div>
This is a design pattern to implement roles like Chef and to separate class parameters into a parameter hierarchy also like chef.
Chef has two very useful features:<br/><br/>
1. <a href="http://docs.opscode.com/essentials_roles.html">Roles</a> <br/>
A server node can have a role like say webserver. Then the cookbooks and recipes ( modules and classes in Puppet terminology) that make up a webserver are related
to the role rather that the server node. <br/> <br/>
2. <a href="http://docs.opscode.com/essentials_cookbook_attribute_files.html">Attribute Precedence.</a> <br/>
Attributes (class Parameters in Puppet terminology) are kept separate from the cookbook (class) code and are applied in a hierarchy
of cookbook,recipe,node,role etc. <br/> <br/>
This has several advantages:<br/>
1. Servers are defined by their roles not their hostnames.<br/>
2. Data is separated from code.<br/>
3. Easier to write and use generic library modules and classes from the puppet forge.<br/>
4. Easier to view and change the parameters of a role<br/>
5. Ability to have different parameters for different environments (test, Prod etc).<br/><br/>
I have build a design pattern called <a href="https://github.com/neillturner/puppet_roles_hiera_parameters">puppet_roles_hiera_parameters</a> to implement these in puppet.<br/> <br/>
Puppet 3.3 incorporate hiera, a hierarchical keyword value store which is very useful to implement a separation of class parameter values
from the class module code. <br/><br/>
The roles can be either set on the individual server or on the puppetmaster in the hieradata.<br/>
If set in the puppetmaster hieradata this takes precedence.<br/><br/>
<b>a. TO SET THE ROLES ON EACH INDIVIDUAL SERVER - MASTERLESS PUPPET:</b><br/><br/>
<b>export FACTER_role_name1=base</b><br/>
<b>export FACTER_role_name2=webserver</b><br/>
puppet apply --modulepath ./modules manifests/site.pp<br/> <br/>
<b>b. TO SET THE ROLES ON THE PUPPETMASTER IN THE HIERADATA:</b><br/> <br/>
on the puppetmaster set the roles in the nodes/<hostname>.yaml file using values<br/>
role::role_name1 to role::role_name4<br/>
for example for a server with a host name of pb65test create a file hieradata/nodes/pb65test.yaml<br/>
<b>----</b><br/>
<b>role::role_name1: 'base'</b><br/>
<b>role::role_name2: 'webserver'</b><br/> <br/><br/>
For more details see the role module on <a href="http://forge.puppetlabs.com/neillturner/role">Puppet Forge</a><br/><br/>
References<br/>
<a href="https://github.com/neillturner/puppet_roles_hiera_parameters">Github - puppet_roles_hiera_parameters</a><br/>
<a href="http://www.craigdunn.org/2012/05/239/">roles and profiles</a><br/>
<a href="http://www.antoncohen.com/2013/06/role-based-puppet-with-hiera.html">Role-based Puppet with Hiera</a><br/>
<a href="https://github.com/antoncohen/vagrant-roles-demo">Github - vagrant-roles-demo</a><br/>
<a href="https://github.com/monokrome/puppet-roles">Github - puppet-roles</a><br/>
<a href="http://docs.opscode.com/essentials_roles.html">About Roles</a> <br/>
<a href="http://blog.brattyredhead.com/blog/2011/11/07/puppet-for-chef-users/">Puppet for chef users</a><br/>
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-21734447297882265182012-05-25T02:03:00.002-07:002013-04-15T08:12:11.363-07:00JMeter load testing in Amazon AWS EC2<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHPxTfSMiboFInWCy5fjr9jtDqH0tKBu3ofYTz2Ukr_wXm5CiIUMrN1Je3ZZ5CHG9mySACQs8qTrMv-yYpUMTomgU9pK1CVC7wG33fCEBcBOce6Esv4KJTlYS4dT3IOLJP_6jg9hcRPkM/s1600/443174_beyond_ship_gauges.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHPxTfSMiboFInWCy5fjr9jtDqH0tKBu3ofYTz2Ukr_wXm5CiIUMrN1Je3ZZ5CHG9mySACQs8qTrMv-yYpUMTomgU9pK1CVC7wG33fCEBcBOce6Esv4KJTlYS4dT3IOLJP_6jg9hcRPkM/s200/443174_beyond_ship_gauges.jpg" width="200" /></a></div>
<a href="http://jmeter.apache.org/">JMeter</a> is popular open source load testing software from Apache. <br />
<br />
JMeter is a Java Graphical application. The easiest way to run it in Amazon EC2 is in a windows server so you can easily access it via RDP.<br />
<br />
JMeter has a master/slave architecture where the client is the master and multiple JMeter servers are slaves running tests in a scalable fashion. It is best to run both in Amazon EC2, the master on a large instance so it has the CPU to process the graphs and the slaves on medium instances that have the network capacity to run the tests. <br />
<br />
1. In Amazon EC2 setup 2 security groups jmeter-master and jmeter-slave with port 3389 open for RDP and set group level access open between them. <br />
<br />
2. Pick an Amazon Windows Image. I selected ami-23457f57. At time of writing the latest 64 bit Windows 2008 R2 Server image. <br />
<br />
3. Launch a large instance with the jmeter-master security group. <br />
<br />
4. Logon via RDP and change the Administrator password and turn off the firewall if Windows 2008 Server. <br />
<br />
5. Download Java JDK version 6 or higher and install. <br />
<br />
6. Download the binary package of <a href="http://jmeter.apache.org/download_jmeter.cgi">Apache Jmeter</a> and unzip to install.<br />
<br />
7. Download the <a href="http://code.google.com/p/jmeter-plugins/">Jmeter Plugins</a> and unzip and copy the JMeterPlugins.jar file to [JMETER_HOME]/lib/ext<br />
<br />
8. Edit the jmeter.property file in [JMETER_HOME]/bin folder<br />
a. uncomment and set<br />
jmeter.save.saveservice.thread_counts=true<br />
b. edit the “remote_hosts” property and set to <br />
slave1:1099,slave2:1099,slave3:1099,slave4:1099<br />
<br />
9. Edit the C:\Windows\System32\drivers\etc\hosts and add<br />
127.0.0.1 slave1<br />
127.0.0.1 slave2<br />
127.0.0.1 slave3<br />
127.0.0.1 slave4 <br />
<br />
10. Add a shortcut on the desktop. Before we run we update with the ip addresses of the slave servers.<br />
<br />
11. Start JMeter by running [JMETER_HOME]/bin/jmeter.bat and create a test script. There are many tutorials on JMeter, for example <a href="http://jmeter.apache.org/usermanual/build-web-test-plan.html">http://jmeter.apache.org/usermanual/build-web-test-plan.html</a>. Also use the JMeter plugins to provide extra graphs. <br />
<br />
12. Create an image of the server. <br />
<br />
13. Start up a slave machine by running an instance using the image built and use the security group jmeter-slave. <br />
<br />
14. update the hosts file on the jmeter-master server with the private IP address of the slave. <br />
<br />
15. Logon to the slave machine via RDP and run [JMETER_HOME]/bin/jmeter-server.bat<br />
<br />
16. On the jmeter-master server in JMeter select run -> remote start -> slave1 to start the test you should seem a message about starting the test.<br />
<br />
17. Enjoy the graphs....change the tasks....add extra slaves......<br />
<br />
<b>References</b><br />
<br />
<a href="http://nesj.net/blog/2011/10/jmeter-remote-testing-on-aws-ec2/">JMeter remote testing on aws ec2</a><br />
<a href="http://vedovini.net/2009/08/jmeter-distributed-testing-with-amazon-ec2/">JMeter distributed testing with amazonec2</a><br />
<a href="http://code.google.com/p/jmeter-plugins/">JMeter Plugins</a><br />
<a href="http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf">JMeter Distributed Testing Step-by-step</a>Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com3tag:blogger.com,1999:blog-8182851079738630242.post-49483940685704152082012-03-22T06:18:00.001-07:002012-04-23T02:03:55.096-07:00JBoss Clustering in Amazon EC2<div class="separator" style="clear: both; text-align: right;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5JfYb6ieSbg7syO2ctFNNvcHpE_0o1LnRDx7or8Jp7P3rg3vZ2sEbUNlnllxfpOOauf8qDx_F47FC4TxCMdcADF2sxaFWCvbuRJoW9DyPUaMvXpmHhu4SKa35a6hpNOFqC2idSRWkdxE/s1600/1382435_teamwork_concept.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5JfYb6ieSbg7syO2ctFNNvcHpE_0o1LnRDx7or8Jp7P3rg3vZ2sEbUNlnllxfpOOauf8qDx_F47FC4TxCMdcADF2sxaFWCvbuRJoW9DyPUaMvXpmHhu4SKa35a6hpNOFqC2idSRWkdxE/s200/1382435_teamwork_concept.jpg" width="200" /></a></div>
Amazon AWS does not allow multicasting so the default
clustering configuration for JBoss needs configuration. Here are the steps to
configure 2 JBoss nodes of a cluster that will run on AWS. <br />
<br />
1. Create 2 security groups JBoss1 and JBoss2 that have group access to each other. <br />
<br />
2. Choose a Linux image and launch with security group JBoss1. I used image:<br />
amazon/suse-sles-11-sp1-v1.01.x86_64.<br />
<br />
3. Install Java JDK 6.<br />
<br />
4. Download JBoss 4.3.2 and unzip:<br />
wget http://sourceforge.net/projects/jboss/files/JBoss/JBoss-4.2.3.GA/jboss-4.2.3.GA-jdk6.zip/download<br />
<br />
5. Download concurrent-1.3.2 jar and copy to $JBOSS_HOME/lib as the gossip server requires it. <br />
<br />
6. In the jboss bin directory create a file gossip.sh to run the gossip server <br />
export JBOSS_HOME=/xxx/jboss-4.2.3.GA<br />
java -cp $JBOSS_HOME/server/all/lib/jgroups.jar:$JBOSS_HOME/lib/commons-logging.jar:$JBOSS_HOME/lib/concurrent-1.3.2.jar org.jgroups.stack.GossipRouter -port 5555 -expiry 30000 -bindaddress jboss1<br />
<br />
7. Edit jboss-4.2.3.GA/server/all/deploy/cluster-service.xml<br />
<br />
replace <br />
<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"<br />
mcast_port="${jboss.hapartition.mcast_port:45566}"<br />
tos="8"<br />
ucast_recv_buf_size="20000000"<br />
ucast_send_buf_size="640000"<br />
mcast_recv_buf_size="25000000"<br />
mcast_send_buf_size="640000"<br />
loopback="false"<br />
discard_incompatible_packets="true"<br />
enable_bundling="false"<br />
max_bundle_size="64000"<br />
max_bundle_timeout="30"<br />
use_incoming_packet_handler="true"<br />
use_outgoing_packet_handler="false"<br />
ip_ttl="${jgroups.udp.ip_ttl:2}"<br />
down_thread="false" up_thread="false"/><br />
<PING timeout="2000"<br />
down_thread="false" up_thread="false" num_initial_members="3"/><br />
with <br />
<UDP ip_mcast="false" mcast_addr="244.0.0.35" mcast_port="45566" ip_ttl="32"<br />
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"/><br />
<PING gossip_host="jboss1" gossip_port="5555"<br />
gossip_refresh="15000" timeout="2000" num_initial_members="2"/><br />
<br />
8. Save the image <br />
<br />
9. Start a second instances using this image and security group jboss2.<br />
<br />
10. On both servers add to the /etc/hosts file the internal ip addresses<br />
For example:<br />
10.248.11.223 jboss1<br />
10.248.10.253 jboss2<br />
<br />
11. Wait until the operating systems updated network addresses by running <br />
ping jboss1 <br />
ping jboss2<br />
<br />
12. On jboss1 start gossip server <br />
./gossip.sh & <br />
<br />
13. On jboss1 started jboss<br />
./run.sh -c all -g TestCluster -b jboss1 <br />
<br />
14. On jboss2 started jboss2<br />
./run.sh -c all g-TestCluster -b jboss2 <br />
<br />
15. You will see messages in the startup that show that the cluster formed. <br />
11:00:59,124 INFO [TestCluster] Dead members: 0 ([])<br />
11:00:59,124 INFO [TestCluster] New Members : 1 ([10.248.10.253:1099])<br />
11:00:59,124 INFO [TestCluster] All Members : 2 ([10.248.11.223:1099, 10.248.10.253:1099])<br />
<br />
16. To automate the setting of the jboss1 and jboss2 ip addresses:<br />
a. Install Ruby if not already installed <br />
<br />
b. install rubygems<br />
wget <a href="http://production.cf.rubygems.org/rubygems/rubygems-1.8.19.tgz" target="_blank">http://production.cf.rubygems.<wbr></wbr>org/rubygems/rubygems-1.8.19.<wbr></wbr>tgz</a><br />
tar xvf rubygems-1.8.19.tgz<br />
cd rubygems-1.8.19<br />
ruby setup.rb<br />
<br />
c. install right_aws amazon interface<br />
gem install right_aws<br />
<br />
d. download scripts and configure<br />
<a href="https://github.com/neillturner/ec2dream/blob/master/scripts/settings.rb">settings.rb</a> <br />
<a href="https://github.com/neillturner/ec2dream/blob/master/scripts/cloud_init.rb">cloud_init.rb</a><br />
<br />
e. create cron job<br />
crontab -e<br />
*/5 * * * * ruby /root/cloud_init.rb generate_hosts=Y <br />
<br />
<b>References</b><br />
<a href="http://www.hugotroche.com/my_weblog/2008/06/clustering-jbos.html">http://www.hugotroche.com/my_weblog/2008/06/clustering-jbos.html</a><br />
<a href="http://xebee.xebia.in/2009/12/03/jboss-clustering-in-5-minutes/">http://xebee.xebia.in/2009/12/03/jboss-clustering-in-5-minutes/</a><br />
<a href="https://community.jboss.org/message/209111">https://community.jboss.org/message/209111</a><br />
<a href="http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html%20">http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html</a><br />
<br />Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com2tag:blogger.com,1999:blog-8182851079738630242.post-2857654575430503462012-02-05T05:43:00.001-08:002019-06-04T06:23:02.615-07:00ProjectsI do traditional on-site consulting and implementation via a negotiable
daily rate as well as remote fixed-priced projects.<br />
<br />
<span style="font-weight: bold;">CLOUD DEVOPS PROJECTS</span><br />
<br />
<a href="http://www.hindawi.com">Hindawi</a><br />
Introduced devops practices to their AWS Windows environment. Introduced Terraform and Packer, documented the existing processes and suggested improvements.<br />
<br />
<a href="http://www.global.com">Global Radio</a><br />
Supported the development team building java microservices using existing devops processes, documented the existing processes and suggested improvements using AWS, Terraform, Linux, Puppet, Jenkins, Java<br />
<br />
<a href="http://www.brave.co.uk">Brave Marketing</a><br />
Responsible for devops setup for an AWS ECS Docker based training application Brave built for Dyson.<br />
<br />
<a href="http://https://www.audionetwork.com">Audio Network</a><br />
Introduced devops to their AWS Windows Server IIS application written in C# using Terraform, Packer, Windows Server, Powershell, ElasticSearch, New Relic<br />
<br />
<a href="http://www.immediate.co.uk">Immediate Media</a><br />
Chef, AWS, Atlas, Terraform Vagrant Puppet to build WordPress platform for websites.<br />
<br />
<a href="http://www.argos.co.uk">Argos</a><br />
Moved their development servers to IBM Softlayer Cloud. Automated the development of Websphere Commerce Server, Websphere Message Broker and DB2 using Test-Kitchen, Chef and Jenkins CI. Dramatically reduced time to create servers and improved quality of builds via automation.<br />
<br />
<a href="http://www.reedbusiness.com/">Reed Business Information</a><br />
Setup Greenfield project for new functionality for Estates Gazettes website using Amazon AWS, Puppet and Test-kItchen. Developers moved from .NET to Nodejs and MongodDB.<br />
Setup Agile development environment around atlassian tools (Jira, Confluence, Stash) and TeamCity for Continuous Integration on Amazon AWS.<br /> Designed and setup the application environment around PM2, Nodejs, Sinopia and MongoDB. Successfully improved the agility of the devlopment team and automated the production deployment process. <br />
<br />
<a href="http://thomascook.com/">Thomas Cook Ecommerce</a><br />
Worked with consultant from McKinsey to improve chef development processes which were error prone. Created a new process around test-kitchen, github, naming standards, separation of library and application cookbooks and implemented a new Tomcat application successfully. <br />
<br />
<a href="https://gds.blog.gov.uk/">Government Digital Service</a><br />
Providing DevOps consultancy to several projects at the UK Govt Home Office using vCloud and Puppet. <br />
<br />
<a href="http://www.pibenchmark.com/">PI Benchmark</a><br />
Created Puppet Architecture for Centos Servers on a Virtual Nutanix platform. <br />
<br />
<a href="http://www.bydeluxe.com/">Deluxe Media Technologies</a><br />
DevOps specialist for a team developing an enterprise media product. Administrating the build environment and platform and helped developers with debugging. Created an administration guide for their product. <br />
<br />
Entrago<br />
Reviewed existing systems and designed chef scripts for the test and production systems for this media startup using Amazon AWS, Linux, Chef, Ruby, Redis, Nodejs, Wowza. <br />
<br />
<a href="http://sky.com/">Sky</a><br />
Built a Jenkins build environment on Amazon AWS for an internet TV cloud project using Amazon AWS, Chef, Ruby, Subversion, Jenkins. <br />
<br />
<a href="http://kcl.ac.uk/">King's College London</a><br />
Reviewed the market for Cloud-based Disaster Recovery Solutions for the College website. Amazon AWS was the lowest cost provider and implemented an Amazon AWS solution. <br />
<br />
OMGO<br />
Setup JBoss Clustering for an Adempiere System on Amazon AWS. <br />
<br />
<a href="http://www.bmj.com/">British Medical Journals</a><br />
Setup Disaster Recovery on Amazon AWS for the BMJ website and applications. This is a multi-tier Java and Oracle application. <br />
<br />
<a href="https://www.my1login.com/">My1Login</a><br />
Migrated the my1Logon website to Amazon EC2 so that they could respond to growth quickly. This is a PHP and MySQL application. <br />
<br />
<a href="http://www.validis.com/">Future Route</a><br />
Migrated
the CreditPal application to Amazon AWS. This saved money compared to
previous hosted solution. This is a multi-tiered Java and MySql
application. <br />
<br />
<br />
<span style="font-weight: bold;"><br />HPC PROJECTS</span><br />
<br />
<a href="http://www.bnpparibas.co.uk">BNP Paribas</a><br />
Supported their large datasynapse hpc grid platform.<br /><br />
<a href="https://www.credit-suisse.com/uk/en/">Credit Suisse</a><br />
Migrated their datasynapse applications to version 5.1. Further project to migrate to Platform Symphony.Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-80589408254235682942009-04-17T07:24:00.012-07:002020-03-22T02:26:36.173-07:00Methodology<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib_2vdj5LX6sIieXy3MO3CV2TrlKWMmvIEXaCDORKhnInqbdEU3kvntPsAa0bxxp1VJIxSfbVM_nNXqmCSJhECsbX-jXI3BbOPL_s4aBGlCUZus80dpuPmEPLLWsy7q3Io0f9PPzFp-d0/s1600/wisdom-cleverness-curve.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5326343264304851410" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib_2vdj5LX6sIieXy3MO3CV2TrlKWMmvIEXaCDORKhnInqbdEU3kvntPsAa0bxxp1VJIxSfbVM_nNXqmCSJhECsbX-jXI3BbOPL_s4aBGlCUZus80dpuPmEPLLWsy7q3Io0f9PPzFp-d0/s320/wisdom-cleverness-curve.png" style="cursor: hand; cursor: pointer; float: right; height: 228px; margin: 0 0 10px 10px; width: 320px;" /></a>
<i>"We used to design infrastructures that support applications. We are now developing new applications that support the cloud platforms".</i>
<ul>
<li>Think of devops as a workflow of tools.</li>
<li>Make sure the tools are loosely coupled so they can be changed easily.</li>
</ul>
Devops consists of 2 parts: infrastructure as code, and deployment pipelines.
<h4>INFRASTRUCTURE AS CODE CONSISTS OF 2 PARTS</h4>
<ul>
<li><b>Orchestrating the Infrastructure</b> (ie defining network, security etc) - Terraform is a good choice. Build it in layers.</li>
<li><b>Configuring Individual Servers.</b> are you going to do immutable or mutable? use packer? use docker containers?
use a config tool like Ansible, chef, puppet or salt?</li>
<li>Both of these need to be stored in <b>Git Repositories.</b> I would start simply with a develop and master branch.</li>
</ul>
<h4>DEPLOYMENT PIPELINES</h4>
<ul>
<li>Typically devs use a CI tool like Jenkins to create their pipelines.</li>
<li>What is the deployable artifact? - zip file, jar file, or docker container.</li>
<li>Where will the artifact be stored? - S3, Artifactory, Dockerhub.</li>
<li>What is the version number strategy for the deployable artifact? - it should end in a build number that increments each build.</li>
<li>The output of the build stage of pipeline is a deployable artifact.</li>
<li>Other stages in the pipelines take the artifact through test, qa, uat and production.</li>
<li>Use a git branch strategy and feature branches named after JIRA tickets.</li>
<li>Use a git tagging strategy to drive the pipeline.</li>
</ul>
<span style="font-weight: bold;">References</span><br />
<a href="https://devopscollective.org/a-plea-for-idempotence-and-immutability/">A Plea for Idempotence and Immutability</a><br />
<a href="https://forrestbrazeal.com/2020/01/05/code-wise-cloud-foolish-avoiding-bad-technology-choices/">Code wise cloud foolish avoiding bad technology choices</a><br />
Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0tag:blogger.com,1999:blog-8182851079738630242.post-57097802973889990212009-03-18T04:28:00.001-07:002019-05-31T07:07:55.231-07:00Contact<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq3cgJ0W7BcTfxwfPtF9h-SnfGQ2GIdUyLJzKq_d0cU3BZmUYzSCVqBbutmNovdOt-ZW-CEbWJdtdpZBHAFZT9zEajhI9hpAEF3Eca4XBsGvp5CyDMUocdsrW_uyzZtTw86ThNlx88dv4/s1600/image2.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq3cgJ0W7BcTfxwfPtF9h-SnfGQ2GIdUyLJzKq_d0cU3BZmUYzSCVqBbutmNovdOt-ZW-CEbWJdtdpZBHAFZT9zEajhI9hpAEF3Eca4XBsGvp5CyDMUocdsrW_uyzZtTw86ThNlx88dv4/s320/image2.JPG" width="320" height="319" data-original-width="549" data-original-height="547" /></a></div>
I enjoy working with developers using agile methodologies, designing server configuration, cloud architectures, build environments and administration of development platforms.<br /><br />
I build open source tools for devops. I've written a number of test kitchen plugins the most popular being <a href='http://github.com/neillturner/kitchen-ansible'>kitchen-ansible</a> and <a href='http://github.com/neillturner/kitchen-puppet'>kitchen-puppet</a> and contribute to <a href="https://registry.terraform.io/modules/neillturner">terraform registry</a> the <a href='http://forge.puppetlabs.com/neillturner'>Puppet Forge</a>.<br /><br />
I've broad experience in the full technology stack:<br />
- Mostly Amazon AWS but also Openstack, Softlayer and vCloud.<br />
- Terraform and Cloud Formation.<br />
- Chef, Puppet, Ansible, Salt and Packer<br />
- Continous Integration with Jenkins and Teamcity.<br />
- Centos and Ubuntu Linux and Windows Server.<br />
- Test tools like test-kitchen, rspec and server-spec.<br />
- Build environments like Teamcity and Jenkins.<br />
- Jira and Confluence tools.<br />
- Oracle. MySql, Redis, MongoDB Databases.<br />
- Tomcat Nodejs Servers.<br />
- Apache, IIS, and Nginx web servers.<br />
- Certified java developer and a background in working for enterprise software vendors.<br />
<br />
I am available for on-site fixed-term contracting via daily rate for Cloud projects and short remote fixed-priced cloud projects and microconsulting.<br />
<br />
<br />
If I can help you please contact me at <span style="font-weight: bold;">neillwturner-at-gmail.com</span>.<br />
<a href="http://www.linkedin.com/in/neillturner"><br /><img alt="View Neill Turner's profile on LinkedIn" border="0" height="15" src="https://www.linkedin.com/img/webpromo/btn_profile_greytxt_80x15.gif" width="80"></a><br />Neill Turnerhttp://www.blogger.com/profile/11967096487210910865noreply@blogger.com0