Skip to content

Latest commit

 

History

History
425 lines (269 loc) · 18.7 KB

1-Development-environment-setup.md

File metadata and controls

425 lines (269 loc) · 18.7 KB

Table of Contents

I. Development environment

1. Project Profile

Address of this project : https://github.com/Mikaelemmmm/go-zero-looklook

The whole project uses go-zero microservices framework, which basically includes go-zero and some middleware developed by related go-zero authors, and the technology stack used is basically the self-developed components of go-zero project team, which is basically go-zero family bucket.

The development environment of this project recommends using docker-compose, using locally connected service, avoiding the complexity of using a service registration and discovery middleware (etcd, nacos, consul, etc.)

Testing, online deployment using Kubernetes a.k.a k8s (also do not need etc.) have a detailed tutorial (build + deployment), you can enter the go-zero community group communication, very easy

The project directory structure is as follows:

.
├── app               -> All business code contains api, rpc and mq (message queue, delay queue, timed tasks)
├── common            -> Generic components error, middleware, interceptor, tool, ctxdata, etc
├── data              -> Contains the data generated by the dependant service (mysql, ES, Redis, etc.). This folder should be git ignored
├── deploy
│   ├── filebeat      -> Docker deployment filebeat configuration
│   ├── go-stash      -> go-stash configuration
│   ├── goctl         -> The goctl template used in the project, copy to $HOME/.goctl can be customized
│   ├── nginx         -> Nginx gateway
│   ├── prometheus    -> Prometheus service configuration
│   └── script
│       ├── gencode   -> Generate api, rpc, and create kafka statements, copy and paste to use
│       └── mysql     -> Generate models for SH tool
├── doc               -> Project documentation
└── modd.conf         -> modd hot-loading configuration file

Check https://github.com/cortesi/modd, for more information about modd tool wa are using to hot-reload the project uppon file change.

*PS: The project will only golang-1.17.7-alpine as base image for the installed modd, if you want to put other tools (goctl, protoc, golint, protoc, golint, etc.) do not use my mirror directly.

2. Use to technology stack

Tips : If you are not familiar with many of the technology stack here, do not be afraid, as long as you will mysql, redis can start these two middleware in the start of the project first run up the project, the other can slowly learn.
  • Kubernetes (a.k.a k8s)
  • go-zero
  • Nginx server (as a Gateway)
  • Filebeat
  • Kafka
  • Go-stash
  • Elasticsearch
  • Kibana
  • Prometheus
  • Grafana
  • Jaeger
  • Go-queue
  • Asynq
  • Asynqmon
  • DTM
  • docker
  • docker-compose
  • mysql
  • redis
  • modd
  • jenkins
  • gitlab
  • harbor

3. Project Architecture Diagram

gozerolooklook

4. Business Architecture Diagram

gozerolooklook

5. Project Environment Setup

⚠️ If you encounter problems during the build process, you can see "9. Common Errors"

The project uses modd hot-loading function instantly modify the code in time to take effect, and do not need to restart each time, change the code automatically in the container reload, local services do not need to start, locally installed sdk is to write code automatically prompted to use, the actual run is since the container lyumikael/go-modd-env:v1.0.0 golang environment. So use goland, vscode are the same

Translated with www.DeepL.com/Translator (free version)

[Note] Since this project has more middleware, starting docker on non-linux may consume more memory, so it is recommended that the memory allocated to docker on the physical machine be adjusted to 8G.

5.1. clone code & update dependencies

git clone https://github.com/Mikaelemmmm/go-zero-looklook
go mod tidy

5.2. The environment on which the project is launched

docker-compose -f docker-compose-env.yml up -d

5.3. Importing Data

5.3.1. Creating a kafka topic

The system uses 3 topics, the default is not to allow the program to automatically create a topic, into the kafka container to create 3 topics

Enter the container

docker exec -it kafka /bin/sh
cd /opt/kafka/bin/

Create 3 topics

./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 -partitions 1 --topic looklook-log
./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 -partitions 1 --topic payment-update-paystatus-topic

looklook-log : The log collection uses the

payment-update-paystatus-topic : Payment success notification

send-wx-mini-tpl-message:Send WeChat applet notifications

5.3.2. Importing mysql data

For local tools to connect to mysql, you need to enter the container first and set remote connection privileges for root

$ docker exec -it mysql mysql -uroot -p
#input password:PXDN93VRKUm8TeE7
$ use mysql;
$ update user set host='%' where user='root';
$ FLUSH PRIVILEGES;

Create database looklook_order && import deploy/sql/looklook_order.sql data

Create database looklook_payment && import deploy/sql/looklook_payment.sql data

Create database looklook_travel && import deploy/sql/looklook_travel.sql data

Create database looklook_usercenter && import looklook_usercenter.sql data

5.4. View Service Environment

Elastic search: http://127.0.0.1:9200/ （⚠️ :This startup time is a bit long）

jaeger: http://127.0.0.1:16686/search (⚠️ :If it fails, rely on es, because es start time is long this may timeout, wait for es start play restart a)

go-stash : If you find that the logs are not collected when kibana clicks next and your kafka gets the data, please restart go-stash and wait for a minute. (⚠️ :If you are mac m1 or linux arm, please change the go-stash image in docker-compose-env.yml kevinwan/go-stash:1.0-arm64, the default is for linux amd)

asynq （Delayed tasks, sheduler tasks, message queues）: http://127.0.0.1:8980/

kibana : http://127.0.0.1:5601/

Prometheus: http://127.0.0.1:9090/

Grafana: http://127.0.0.1:3001/ ， The default account and password are admin

Mysql : Self-client tools (Navicat, Sequel Pro) to view

  • host : 127.0.0.1

  • port : 33069

  • username : root

  • pwd : PXDN93VRKUm8TeE7

Redis : View by tool (redisManager)

  • host : 127.0.0.1

  • port : 36379

  • pwd : G62m50oigInC30sf

Kafka: (pub.sub)Self-client tool view

  • host : 127.0.0.1

  • port : 9092

5.5. Start Service

5.5.1. Pull the runtime environment image

Because this project is using docker + hot-loading, that is, the change is effective

All API + RPC services under frontend app use modd + golang

Direct docker-compose to start can, but consider the dependencies may be relatively large, will affect the start of the project, so it is best to pull down this image before starting the project

docker pull lyumikael/gomodd:v1.0.0 #This is used by all API+RPC startup services under the app, if you are "mac m1" : lyumikael/go-modd-env:v1.0.0

Note: If you add new services under the app, remember to add a copy of modd.conf in the root directory of the project.

About modd more usage can go here to learn: https://github.com/cortesi/modd, the project image is only golang-1.17.7-alpine as the base image installed modd in the internal.

If you want to add goctl, protoc, golint, etc., do not use my image directly to create a mirror is the same ha

5.5.2. Start Project

docker-compose up -d

[Note] The dependency is on the docker-compose.yml configuration in the project root directory

5.6. View project operation

Visit http://127.0.0.1:9090/, click on the menu above "Status", click on Targets, the blue one is started, the red one is not started successfully

Note] If it is the first time to pull the project, each project container for the first time to build pull dependencies, this depends on the network situation, may be slightly slower to have the service, this is normal, if you encounter the project does not start up, such as order-api , manually in order-api code to write something randomly to save a trigger to recompile to see the log on it

docker-compose logs -f

You can see that the prometheus also shows success, and similarly the other also row once, start successfully on it

image-20220120103641110.png

7. Access Program

Since we use nginx as the gateway, the nginx gateway is configured in docker-compose, which is also configured in docker-compose, nginx exposes port 8888 to the public, so we access through port 8888

$ curl  -X POST "http://127.0.0.1:8888/usercenter/v1/user/register" -H "Content-Type: application/json" -d "{\"mobile\":\"18888888888\",\"password\":\"123456\"}"

response:
{"code":200,"msg":"OK","data":{"accessToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzM5NjY0MjUsImlhdCI6MTY0MjQzMDQyNSwiand0VXNlcklkIjo1fQ.E5-yMF0OvNpBcfr0WyDxuTq1SRWGC3yZb9_Xpxtzlyw","accessExpire":1673966425,"refreshAfter":1658198425}}

Note] If the access to nginx fails and the access success can be ignored, it may be that nginx depends on the back-end services, before the back-end services did not start up, nginx did not start up here, restart once nginx can, the project root directory restart

docker-compose restart nginx

6. Log Collection

Collect project logs to es (filebeat collects logs -> kafka -> go-stash consumes kafka logs -> output to es, kibana views es data)

Visit kibana http://127.0.0.1:5601/ and create log index

Click on the top left menu (the three horizontal lines), find Analytics - > click on discover

image-20220120105829870

Then on the current page, Create index pattern->type looklook-* -> Next Step -> select @timestamp->Create index pattern

Then click the top left menu, find Analytics->click discover, the logs are displayed (if not, check filebeat, go-stash, use docker logs -f filebeat to view)

image-20220120105947733

⚠️ Common reasons for collection failure

  • There is no topic in kafka to create a log: looklook-log

    Solution: Go to kafka and create looklook-log, restart filebeat, go-stash

  • started after go-stash

    Solution: restart go-stash once

  • The go-stash image is being used incorrectly

    Look at the go-stash log, if there is a core dumped, it means that the mirror is being used incorrectly.

    Solution:

    If you are mac m1 or linux arm, please change the go-stash image in docker-compose-env.yml kevinwan/go-stash:1.0-arm64, the default is for linux amd.

  • docker version problem

    Answer:

    This I did not actually encounter, but some students using docker version is 1.13 encountered, filebeat configuration file configuration to collect docker path low version docker may not be the same location resulting in the collection of docker internal logs, it is best to upgrade the docker18.03.1 above the current no problem, 17 did not actually test , 17 My side of the docker version is Version: 20.10.8

  • Internal kafka problem

    Solution:

    1. docker logs check the container logs of kafka, filbeat, go-stash, es in order to make sure the services are OK

    2. first docker logs -f filebeat to see if the filebeat is correctly connected to kafka

    3. Go into the kafka container and execute consume kafka-log messages to see if the filebeat messages have been sent to kafka

    docker exec -it kafka /bin/sh
cd /opt/kafka/bin
./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic looklook-log

    Note] If you can consume the messages, it means that filebeat and kafka are fine, so go check go-stash, es

    If you can not consume

    1. it should be a connection problem between filebeat and kafka, to see if the kafka configuration information Listen is modified

    2. In the kafka container internal command line using consumer.sh consumption looklook-log, another terminal command line with producer.sh to looklook-log send messages, if the consumer can not receive, indicating that the kafka problem, docker logs -f kafka to see what the problem

7. Introduction of this project mirror

  • nginx : gateway （nginx->API->RPC
  • cosmtrek/air.modd : Our business code development depends on the environment image, the reason why we use this is because air hot-loading, writing code compiled in real time is too convenient, this image is air + golang, in fact, after we start our own business services, our business services are running in this image
  • wurstmeister/kafka : kafka
  • wurstmeister/zookeeper :kafka dependencies on zookeeper
  • redis:Business use of redis
  • mysql:Databases used by the business
  • prom/prometheus:Monitoring Business
  • grafana/grafana :The prometheus ui is hard to read and is used to display the data collected by prometheus
  • elastic/filebeat : Collecting logs to kafka
  • go-stash : Consume logs in kafka, desensitize, filter and export to es
  • docker.elastic.co/elasticsearch/elasticsearch : Storage of collected logs
  • docker.elastic.co/kibana/kibana :Show elasticsearch
  • jaegertracing/all-in-one:Link Tracking
  • go-stash : After filebeat collects the logs to kafka, go-stash goes to consume kafka to desensitize the data, filter the content in the logs, and finally output to es

8. Project Development Proposal

  • app: decentralizes all business service codes

  • common: Put up a public base library of all services

  • data:The data project depends on the data generated by the middleware, so in practice you should ignore this directory and the data generated in this directory in git

  • Generate API, RPC code:

Usually when we generate api, rpc code manually to knock goctl command is rather long and can not remember, so we go directly to deploy/script/gencode/gen.sh to copy the code. For example, I added a new service in usercenter service, change the password, after writing the api file, go to usercenter/cmd/api/desc directory, copy the generate api command in deploy/script/gencode/gen.sh to run it

goctl api go -api *.api -dir ../  -style=goZero

After writing the proto file, just copy the generate rpc command from deploy/script/gencode/gen.sh and run it

goctl >= 1.3 Go to the "services/cmd/rpc/pb" directory and execute the following command

goctl rpc protoc *.proto --go_out=../ --go-grpc_out=../  --zrpc_out=../
sed -i "" 's/,omitempty//g' *.pb.go

goctl < 1.3 Enter the "service/cmd" directory and execute the following command

goctl rpc proto -src rpc/pb/*.proto -dir ./rpc -style=goZero
sed -i "" 's/,omitempty//g'  ./rpc/pb/*.pb.go

Note] It is recommended that when generating the rpc file, execute the following command once more to remove the omitempty generated by protobuf, otherwise the field will not be returned if it is nil

  • Generate kafka code:

    Because the project uses go-queue kq to do the message queue, kq and depend on the kafka, the actual is the use of kafka to do the message queue, but kq default is required in advance we build the topic, not allowed to automatically generate the default, so the command is also ready, directly copy the deploy/script/gencode/gen.sh to create the kafka topic code

     kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 -partitions 1 --topic {topic}

  • To generate the model code, run deploy/script/mysql/genModel.sh directly with the parameters

  • API project in the .api file we did a split, unified into the desc folder of each API, because if all the content is written in the API may not be easy to view, so do a split, write all the methods to an API, other entities and req, rep unified into a folder defined separately is clearer

  • Generate mod, error handling when using the template redefinition, the project used a custom goctl template in the project deploy/goctl under

9. Common errors

  1. Grafana reports an error at creation stage, start docker-compose-env.yml container
  1. Issue when starting filebeat:
  • Error: filebeat container startup error Exiting: error loading config file: config file ("filebeat.yml") must be owned by the user identifier (uid=0) or root
  • Solution: Due to different file owners (I clone the project under normal user), the config file of filebeat must be owned by root, you need to run sudo chown root deploy/filebeat/conf/filebeat.yml
  1. Issue when starting Elasticsearch:
  • Error: elasticsearch container startup error `ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes] ; Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes``
  • Solution: The reason for the error is that es does not have permission to operate the mount directory and cannot bind the nodes, solution, modify the permissions sudo chmod 777 data/elasticsearch/data (I don't know which user started es, so I changed 777 the files permission)
  1. Jaeger depends on elasticsearch and does not fail to restart automatically

10. Follow-up

Due to the project as it relates to the technology stack is slightly more, will gradually add a little bit by chapter