iBrokeTheCode commited on
Commit
9c639a1
·
1 Parent(s): 459b8f5

chore: Add LESSONS file

Browse files
Files changed (1) hide show
  1. LESSONS.md +342 -0
LESSONS.md ADDED
@@ -0,0 +1,342 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Lessons Learned
2
+
3
+ ## Table of Contents
4
+
5
+ - [Docker](#1-docker)
6
+ - [Dev Containers](#2-dev-containers)
7
+ - [Redis](#3-redis)
8
+ - [Postgres](#4-postgres)
9
+ - [Unit Tests](#5-unit-tests)
10
+ - [Locust](#6-locust)
11
+ - [AWS](#7-aws)
12
+ - [GitHub Actions](#8-github-actions)
13
+
14
+ ## 1. Docker
15
+
16
+ - Install [Docker](https://docs.docker.com/engine/install/)
17
+ - Some basic Docker commands:
18
+
19
+ ```bash
20
+ # List all containers
21
+ docker ps
22
+
23
+ # List all images
24
+ docker images
25
+
26
+ # Remove resources
27
+ docker rmi $(docker images -q)
28
+ docker rm $(docker ps -aq)
29
+
30
+ # Network
31
+ docker network ls
32
+ docker network rm <network_id>
33
+
34
+ # Volumes
35
+ docker volume ls
36
+ docker volume rm <volume_id>
37
+
38
+ # Build an image
39
+ docker build -t <image_name> .
40
+
41
+ # Build a test image (target)
42
+ docker build -t ui_test --progress=plain --target test .
43
+
44
+ # Run a container
45
+ docker run -p <port>:<port> -d <image_name>
46
+ ```
47
+
48
+ ### 1.1 Dockerfile
49
+
50
+ - A basic example of a Dockerfile
51
+
52
+ ```Dockerfile
53
+ # Base image
54
+ FROM python:3.9
55
+
56
+ # Set working directory
57
+ WORKDIR /app
58
+
59
+ # Copy files
60
+ COPY . /app
61
+
62
+ # Install dependencies (during build time)
63
+ RUN pip install -r requirements.txt
64
+
65
+ # Run the app (after installing dependencies)
66
+ CMD ["python", "main.py"]
67
+ ```
68
+
69
+ ### 1.2 Docker Compose
70
+
71
+ - A basic example of a Docker Compose file running microservices:
72
+
73
+ ```yml
74
+ services:
75
+ api: # Name of the service
76
+ build: # Build the image
77
+ context: ./ui
78
+ target: build # Target
79
+ ports: # Ports
80
+ - "8000:8000"
81
+ environment: # Env variables
82
+ POSTGRES_DB: $POSTGRES_DB
83
+ POSTGRES_USER: $POSTGRES_USER
84
+ POSTGRES_PASSWORD: $POSTGRES_PASSWORD
85
+ DATABASE_HOST: $DATABASE_HOST
86
+ depends_on: # Dependencies
87
+ - redis
88
+ - db
89
+ networks: # Network
90
+ - shared_network
91
+ volumes: # Volumes
92
+ - ./uploads:/src/uploads
93
+ db: # Another service
94
+ image: postgres:13-alpine
95
+ volumes: # Volumes
96
+ - postgres_data:/var/lib/postgresql/data
97
+ redis: # Another service
98
+ image: redis:6.2.6
99
+ networks:
100
+ - shared_network
101
+ networks:
102
+ shared_network:
103
+ volumes:
104
+ postgres_data:
105
+ ```
106
+
107
+ ## 2. Dev Containers
108
+
109
+ - Install [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension
110
+ - Create a `.devcontainer` folder
111
+ - Create folders for each service you want to develop (api, model, ui, etc.)
112
+ - Create a `devcontainer.json` file
113
+ - Optionally you can create Dockerfiles or Docker Compose files for each service
114
+
115
+ ```json
116
+ {
117
+ "name": "ML Project - API",
118
+ "dockerComposeFile": "../../docker-compose-dev.yml",
119
+ "service": "api",
120
+ "workspaceFolder": "/src",
121
+ "customizations": {
122
+ "vscode": {
123
+ "extensions": ["ms-python.python"]
124
+ }
125
+ },
126
+ "shutdownAction": "none"
127
+ }
128
+ ```
129
+
130
+ - Note: Don't use `COPY` command inside your Dockerfile, use `volumes` in your Docker Compose file to edit files inside the container. (for development stage)
131
+
132
+ ```yml
133
+ # ...
134
+ volumes:
135
+ - ./ui:/src:cached
136
+ # ...
137
+ ```
138
+
139
+ - Run the dev container by selecting `devcontainer: Reopen in Container` option (`ctrl + shift + P`)
140
+
141
+ ## 3. Redis
142
+
143
+ - To connect to a Redis instance by terminal you can use the following command:
144
+
145
+ ```bash
146
+ redis-cli
147
+ ```
148
+
149
+ - Some basics commands:
150
+
151
+ ```bash
152
+ # Set a key
153
+ SET key value
154
+
155
+ # Get a key
156
+ GET key
157
+
158
+ # Delete a key
159
+ DEL key
160
+
161
+ # List all keys
162
+ KEYS *
163
+ ```
164
+
165
+ Review [Redis Commands Cheat Sheet](https://redis.io/learn/howtos/quick-start/cheat-sheet) for more commands.
166
+
167
+ - Monitor the Redis instance with the command:
168
+
169
+ ```bash
170
+ redis-cli monitor
171
+ ```
172
+
173
+ ## 4. Postgres
174
+
175
+ - To connect to a Postgres instance by terminal you can use the following command:
176
+
177
+ ```bash
178
+ psql -U <username> -p 5432 -d <database_name>
179
+ ```
180
+
181
+ - Can also use a Postgres GUI client, like DBeaver. So you can use the connection values from the Docker Compose file.
182
+
183
+ - Some basics commands are:
184
+
185
+ ```bash
186
+ # List all databases
187
+ \l
188
+
189
+ # Create a database
190
+ CREATE DATABASE <database_name>;
191
+
192
+ # List all tables
193
+ \dt
194
+
195
+ # Exit
196
+ \q
197
+ ```
198
+
199
+ ## 5. Unit Tests
200
+
201
+ ### 5.1 Unittest
202
+
203
+ - To run unit tests you can use the following commands:
204
+
205
+ ```bash
206
+ # Run test file
207
+ python3 -m unittest -vvv tests.test_model
208
+
209
+ # Run an individual test
210
+ python -m unittest -vvv tests.test_image_classifier_app.TestMLService.test_login_failure
211
+ ```
212
+
213
+ - Additionally can run test from the python module:
214
+
215
+ ```py
216
+ if __name__ == "__main__":
217
+ unittest.main(verbosity=2)
218
+ ```
219
+
220
+ And then run the module
221
+
222
+ ```bash
223
+ python tests/test_image_classifier_app.py
224
+ ```
225
+
226
+ ### 5.2 Pytest
227
+
228
+ - To run unit tests you can use the following commands:
229
+
230
+ ```bash
231
+ # Run test file
232
+ pytest -v -s tests/test_model.py
233
+
234
+ # Run an individual test
235
+ pytest -v -s tests/test_image_classifier_app.py::TestMLService::test_login_failure
236
+ ```
237
+
238
+ ## 6. Locust
239
+
240
+ - Install [locust](https://docs.locust.io/en/stable/quickstart.html#locust-s-web-interface)
241
+
242
+ ```bash
243
+ uv add locust
244
+ ```
245
+
246
+ - Create a folder `stress_test` and `locustfile.py` file
247
+
248
+ ```py
249
+ # Basic example
250
+ from locust import HttpUser, task
251
+
252
+ class HelloWorldUser(HttpUser):
253
+ @task
254
+ def hello_world(self):
255
+ self.client.get("/hello")
256
+ self.client.get("/world")
257
+ ```
258
+
259
+ - Run `locust -f stress_test/locustfile.py`
260
+ - Open `http://127.0.0.1:8089`
261
+ - Start a load test and fill the number of users/ramp up
262
+ - Add the host, for example `http://localhost:8000`
263
+ - Start the load test
264
+ - Review the results, stats and charts
265
+
266
+ ## 7. AWS
267
+
268
+ - Download the `epm` file from your AWS account
269
+ - Give read permissions to the file with:
270
+ ```bash
271
+ chmod 400 file.epm
272
+ ```
273
+ - Connect it via ssh
274
+ ```bash
275
+ ssh -i file.epm <ec2_user>@<public_ip>
276
+ ```
277
+ - To copy files from local host to remote host:
278
+
279
+ ```bash
280
+ scp -i file.epm -r <local_path> <ec2_user>@<public_ip>: # Default home
281
+ scp -i file.epm -r <local_path> <ec2_user>@<public_ip>:<remote_path>
282
+ ```
283
+
284
+ - Create a tunnel to the remote host:
285
+
286
+ ```bash
287
+ ssh -L <local_port>:<remote_host>:<remote_port> -i file.epm <ec2_user>@<public_ip>
288
+ ```
289
+
290
+ ## 8. GitHub Actions
291
+
292
+ GitHub Actions is a **CI/CD (Continuous Integration and Continuous Deployment)** tool built into GitHub that allows you to **automate workflows** directly from your repository. With Actions, you can run tests, build your code, deploy applications, and perform other automated tasks whenever certain events occur (like pushing code, creating pull requests, or publishing releases).
293
+
294
+ Key points:
295
+
296
+ - **Event-driven**: Workflows run based on triggers like `push`, `pull_request`, or scheduled cron jobs.
297
+ - **YAML-based**: Workflows are defined in `.github/workflows/` using YAML syntax.
298
+ - **Cross-platform**: Supports Linux, Windows, and macOS runners.
299
+ - **Marketplace**: Offers reusable actions to speed up development.
300
+
301
+ ---
302
+
303
+ ### **Basic Example: CI Workflow for Python Project**
304
+
305
+ ```yaml
306
+ # File: .github/workflows/python-ci.yml
307
+ name: Python CI
308
+
309
+ on:
310
+ push:
311
+ branches: ["main"]
312
+ pull_request:
313
+ branches: ["main"]
314
+
315
+ jobs:
316
+ build:
317
+ runs-on: ubuntu-latest
318
+
319
+ steps:
320
+ - name: Checkout code
321
+ uses: actions/checkout@v3
322
+
323
+ - name: Set up Python
324
+ uses: actions/setup-python@v4
325
+ with:
326
+ python-version: "3.10"
327
+
328
+ - name: Install dependencies
329
+ run: |
330
+ python -m pip install --upgrade pip
331
+ pip install -r requirements.txt
332
+
333
+ - name: Run tests
334
+ run: |
335
+ pytest
336
+ ```
337
+
338
+ **What this does:**
339
+
340
+ - Runs when you **push or create a pull request** to the `main` branch.
341
+ - Uses **Ubuntu** as the environment.
342
+ - Sets up **Python 3.10**, installs dependencies, and runs **pytest** for tests.