Skip to content

Commit f27029a

Browse files
author
Darryl Osborne
committed
Update lustre tutorial
1 parent e3468ee commit f27029a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+2230
-2
lines changed

‎.DS_Store‎

2 KB
Binary file not shown.

‎lustre/.DS_Store‎

14 KB
Binary file not shown.

‎lustre/01-create-environment/readme.adoc‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
= Create Tutorial Environment
2+
:toc:
3+
:icons:
4+
:linkattrs:
5+
:imagesdir: ../resources/images
6+
7+
8+
== Summary
9+
10+
This section will create a new environment (see diagram below) that you will use for the tutorial.
11+
12+
== Duration
13+
14+
NOTE: It will take approximately 15 minutes for CloudFormation to automatically create your tutorial environment, after completing the *Step-by-step Guide* below.
15+
16+
== Diagram
17+
18+
image::fsx-lustre-tutorial-environment-architecture.png[align="left", width=600]
19+
20+
21+
== Step-by-step Guide
22+
23+
=== CloudFormation
24+
25+
IMPORTANT: Read through all steps below and watch the quick video before *_clicking_* the *Deploy to AWS* button.
26+
27+
image::create-environment.gif[align="left", width=600]
28+
29+
30+
31+
. Click on the *Deploy to AWS* button and follow the CloudFormation prompts to begin.
32+
+
33+
Amazon FSx is currently available in 6 regions.
34+
+
35+
File system creation is currently not supported in the following Availability Zones:
36+
+
37+
[cols="1,1,2", options="header"]
38+
|===
39+
|Region | Availability Zone ID | Mapping
40+
| *N. Virginia* (us-east-1)
41+
| *use1-az3*
42+
| Open the link:https://us-east-1.console.aws.amazon.com/vpc/home?region=us-east-1#subnets:[us-east-1 VPC] console for AZ mapping.
43+
44+
| *Oregon* (us-west-2)
45+
| *usw2-az4*
46+
| Open the link:https://us-west-2.console.aws.amazon.com/vpc/home?region=us-west-2#subnets:[us-west-2 VPC] console for AZ mapping.
47+
48+
| *Sydney* (ap-southeast-2)
49+
| *apse2-az3*
50+
| Open the link:https://ap-southeast-2.console.aws.amazon.com/vpc/home?region=ap-southeast-2#subnets:[ap-southeast-2 VPC] console for AZ mapping.
51+
52+
| *Tokyo* (ap-northeast-1)
53+
| *apne1-az3*
54+
| Open the link:https://ap-northeast-1.console.aws.amazon.com/vpc/home?region=ap-northeast-1#subnets:[ap-northeast-1 VPC] console for AZ mapping.
55+
|===
56+
+
57+
When selecting availability zones in the CloudFormation create stack wizard, take care not to select an availability zone mapped to one of the Availability Zone IDs listed above.
58+
+
59+
TIP: *_Context-click (right-click)_* the *Deploy to AWS* button and open the link in a new tab or window to make it easy to navigate between this github tutorial and AWS Console.
60+
+
61+
|===
62+
|Region | Launch template with a new VPC
63+
| *N. Virginia* (us-east-1)
64+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
65+
66+
| *Ohio* (us-east-2)
67+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
68+
69+
| *Oregon* (us-west-2)
70+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
71+
72+
| *Ireland* (eu-west-1)
73+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
74+
75+
| *Sydney* (ap-southeast-2)
76+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
77+
78+
| *Tokyo* (ap-northeast-1)
79+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
80+
81+
| *Hong Kong* (ap-east-1)
82+
a| image::deploy-to-aws.png[link=https://console.aws.amazon.com/cloudformation/home?region=ap-east-1#/stacks/new?stackName=fsx-lustre-tutorial&templateURL=https://s3.amazonaws.com/amazon-fsx/tutorial/lustre/templates/00-fsx-tutorial.yaml]
83+
|===
84+
+
85+
. Accept the defaults on the *Prerequisite - Prepare template* page and *_click_* *Next*.
86+
+
87+
. Accept the default stack name and *_click_* *Next*. *_Enter_* values for all parameters.
88+
+
89+
[cols="3,10"]
90+
|===
91+
| *VPC CIDR*
92+
a| Select a CIDR that will be used for the VPC.
93+
94+
| *Availability Zones*
95+
a| Select two (2) availability zones for your VPC. An Amazon EC2 instance and the FSx file system will be created in the *first* availability zone.
96+
97+
| *Key Name*
98+
a| Select an EC2 key name. This key name is associated with a key pair used to securely connect (SSH) to the EC2 instance.
99+
100+
| *Email address*
101+
a| Enter the email address that will receive notifications for low free storage capacity CloudWatch alarms. Important! An AWS Notification Subscription Confirmation email will be sent to this email address when the CloudFormation stack is created. You must take the action described in that email to confirm the SNS topic subscription before the email address can start receiving alarm notifications.
102+
|===
103+
+
104+
. After you have entered values for all parameters, *_click_* *Next*.
105+
. *_Accept_* the default values of the *Configure stack options* and *Advanced options* sections and *_click_* *Next*.
106+
. *_Review_* the CloudFormation stack settings.
107+
. *_Click_* both checkboxes in the blue *Capabilities* box at the bottom of the page.
108+
+
109+
image::cloudformation-capabilities.png[align="left", width=420]
110+
+
111+
. *_Click_* *Create stack*.
112+
113+
In approximately 15 minutes an email will be sent to the email address entered above (see sample below). Open the email message and *_click_* the *Confirm subscription* link. This will allow Amazon SNS to send CloudWatch alarm notifications to this email address during the tutorial.
114+
115+
image::aws-notification-subscription-email.png[align="left",width=420]
116+
117+
== Next section
118+
119+
Click the button below to go to the next section.
120+
121+
image::02-examine-data-repository-integration.png[link=../02-examine-data-repository-integration/, align="right",width=420]
122+

‎lustre/02-examine-data-repository-integration/readme.adoc‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
= Examine data repository integration
2+
:toc:
3+
:icons:
4+
:linkattrs:
5+
:imagesdir: ../resources/images
6+
7+
8+
== Summary
9+
10+
This section will examine Amazon FSx for Lustre data repository integration with Amazon S3.
11+
12+
The tutorial environment created an FSx for Lustre file system with a data repository integrated with the "nasanex" bucket in the US West (Oregon) region. link:https://registry.opendata.aws/nasanex/[NASA NEX] is a part of the link:https://registry.opendata.aws/[Registry of Open Data on AWS] project and is a collection of Earth science datasets maintained by NASA, including climate change projections and satellite images of the Earth's surface.
13+
14+
15+
== Duration
16+
17+
NOTE: It will take approximately 15 minutes to complete this section.
18+
19+
20+
== Step-by-step Guide
21+
22+
IMPORTANT: Read through all steps below before continuing.
23+
24+
=== Connect to *Linux Instance 0*
25+
26+
image::connect-linux-instance-0.gif[align="left", width=600]
27+
28+
29+
. Open the link:https://console.aws.amazon.com/ec2/[Amazon EC2] console.
30+
+
31+
TIP: *_Context-click (right-click)_* the link above and open the link in a new tab or window to make it easy to navigate between this github tutorial and Amazon EC2 console.
32+
+
33+
. Make sure you are in the same *AWS Region* where you *_created_* your tutorial environment.
34+
. *_Select_* the EC2 instance created by the CloudFormation template. The name is "*Linux Instance 0*".
35+
. *_Click_* the *Connect* button.
36+
. *_Copy_* the SSH command and *_click_* *Close*.
37+
. *_Open_* your *terminal* application and *_paste_* the SSH command in a terminal window.
38+
. *_Follow_* the prompts to SSH into the instance.
39+
40+
=== Examine *s3://nasanex* data repository integration
41+
42+
*_Copy_*, *_paste_*, then *_execute_* the shell commands below in the SSH terminal session of *Linux Instance 0* to answer the following questions:
43+
44+
. Is the FSx for Lustre file system mounted?
45+
+
46+
[source,bash]
47+
----
48+
mount -t lustre
49+
50+
----
51+
+
52+
. How long does it take to list the entire file system?
53+
+
54+
[source,bash]
55+
----
56+
time lfs find /mnt/fsx
57+
58+
----
59+
+
60+
. What file types did you see?
61+
. How many files?
62+
+
63+
[source,bash]
64+
----
65+
time lfs find /mnt/fsx --type f | wc -l
66+
67+
----
68+
+
69+
. How many directories?
70+
+
71+
[source,bash]
72+
----
73+
time lfs find /mnt/fsx --type d | wc -l
74+
75+
----
76+
+
77+
. How many small files (< 512 KiB)?
78+
+
79+
[source,bash]
80+
----
81+
time lfs find /mnt/fsx --type f --size -512k | wc -l
82+
83+
----
84+
+
85+
. How many large files (> 100 MiB)?
86+
+
87+
[source,bash]
88+
----
89+
time lfs find /mnt/fsx --type f --size +100M | wc -l
90+
91+
----
92+
+
93+
. How many .nc, .hdf, .tif, .gz files?
94+
+
95+
[source,bash]
96+
----
97+
time lfs find /mnt/fsx --type f --name *.nc | wc -l
98+
time lfs find /mnt/fsx --type f --name *.hdf | wc -l
99+
time lfs find /mnt/fsx --type f --name *.tif | wc -l
100+
time lfs find /mnt/fsx --type f --name *.gz | wc -l
101+
102+
----
103+
+
104+
. How much metadata (MDT) has been loaded into the file system?
105+
+
106+
How much data (all the OSTs) has been loaded into the file system?
107+
+
108+
How much data storage capacity is available?
109+
+
110+
[source,bash]
111+
----
112+
time lfs df -h
113+
114+
----
115+
116+
117+
=== Verify your results
118+
119+
The results of your queries should match the following:
120+
121+
[cols="3,10"]
122+
|===
123+
| Query | Results
124+
125+
| Is the FSx for Lustre file system mounted?
126+
| 10.0.1.193@tcp:/fsx on /mnt/fsx type lustre (rw,lazystatfs) (you will have a different IP address)
127+
128+
| How long does it take to list the entire file system?
129+
| ~real 0m47.373s
130+
131+
| What file types did you see?
132+
| .hdf .nc .gz .tif .json .md5 .txt .pdf
133+
134+
| How many files?
135+
| 373572
136+
137+
| How many directories?
138+
| 42242
139+
140+
| How many small files (< 512 KiB)?
141+
| 23692
142+
143+
| How many large files (> 100 MiB)?
144+
| 169617
145+
146+
| How many .nc, .hdf, .tif, .gz files?
147+
| .hdf = 207552; .tif = 11095; .nc = 87002; .gz = 42009
148+
149+
| How much storage is used by the metadata target (MDT)?
150+
| 934.1M
151+
152+
| How much storage is used by all the object storage targets (OSTs)?
153+
| 27.M
154+
155+
| How much data storage capacity is available?
156+
| 6.6 T
157+
|===
158+
159+
== Next section
160+
161+
Click the button below to go to the next section.
162+
163+
image::03-load-data.png[link=../03-load-data/, align="left",width=420]
164+
165+
166+
167+

0 commit comments

Comments
 (0)