Deployment for self-hosting

[ojeytonwilliams]

Chapter should have at least one well documented (and hopefully inexpensive) way to deploy the site.

To that end, I'm going to try to deploy to https://railway.app/ and document the process.

[ojeytonwilliams]

It's been a while, but I've created a template https://railway.app/template/it9kIn. Sorry for the delay, @allella, but it's ready for you test.

The process is still quite involved, unfortunately. What you need to do is the following:

1. Setup an Auth0 account https://auth0.com/
2. Create an Auth0 Application https://manage.auth0.com/ > Applications > Applications > Create Application
3. Navigate to https://railway.app/new/template/it9kIn
4. Fill in the environment variable with placeholder text (they're all required, but it's easier to configure them later)
5. Wait until the Activity panel appears on the right
6. Go to Chapter Client > Settings > Domains > Generate Domain (you can customise the domain it generates)
7. Go to Chapter Server > Settings > Domains > Generate Domain
8. Click on Settings (above the panel) > Shared Variables > production
9. Add AUTH0_AUDIENCE, AUTH0_DOMAIN (from dashboard > Applications > APIs > Auth0 Management API and dashboard -> Applications -> your-application -> Settings, respectively)
10. Add CLIENT_LOCATION with the domain you generated
11. Add USE_AUTH0=true
12. Go to Chapter Client > Variables > Raw Editor and add NEXT_PUBLIC_USE_AUTH0=${{shared.USE_AUTH0}} NEXT_PUBLIC_AUTH0_DOMAIN=${{shared.AUTH0_DOMAIN}} NEXT_PUBLIC_AUTH0_CLIENT_ID=Your client id from the dashboard NEXT_PUBLIC_AUTH0_AUDIENCE=${{shared.AUTH0_AUDIENCE}} NEXT_PUBLIC_SERVER_URL=Your full server url # must be a complete url e.g. https://www.example.com NEXT_PUBLIC_CLIENT_URL=${{shared.CLIENT_LOCATION}} NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT=production
13. Go to Chapter Server > Variables > Raw Editor and add AUTH0_AUDIENCE=${{shared.AUTH0_AUDIENCE}} AUTH0_DOMAIN=${{shared.AUTH0_DOMAIN}} CLIENT_LOCATION=${{shared.CLIENT_LOCATION}} USE_AUTH0=${{shared.USE_AUTH0}} COOKIE_DOMAIN=up.railway.app SESSION_SECRET=SetThisTo32orMoreRandomCharacters UNSUBSCRIBE_SECRET=SetThisTo32orMoreRandomCharacters CHAPTER_EMAIL=Whatever you like <chapter@email.address> SES_ACCESS_KEY_ID= SES_SECRET_ACCESS_KEY= DATABASE_URL=${{Postgres.DATABASE_URL}} PGDATABASE=${{Postgres.PGDATABASE}} PGHOST=${{Postgres.PGHOST}} PGPASSWORD=${{Postgres.PGPASSWORD}} PGPORT=${{Postgres.PGPORT}} PGUSER=${{Postgres.PGUSER}} EMAIL_SERVICE=one of (smtp, sendgrid and ses) EMAIL_HOST= EMAIL_PASSWORD= EMAIL_USERNAME= SENDGRID_KEY= SENDGRID_EMAIL=

Depending on which email service you'd like to use, you need (EMAIL_USERNAME, EMAIL_PASSWORD and EMAIL_HOST), (SENDGRID_KEY) or (SES_ACCESS_KEY_ID and SES_SECRET_ACCESS_KEY). That is for SMTP, Sendgrid and SES respectively.

Configure Auth0

Once this is done and you have your client URL, it should be added to your Auth0 application's Allowed Callback URLs, Allowed Logout URLs and Allowed Web Origins.

Prepare the database

# starting from the root of the repo
# install railway cli locally https://docs.railway.app/develop/cli
railway login
railway link
# connect to the server, not the client:
railway service
# install dependencies
npm i
# run the migration
railway run npm run prisma migrate deploy
# go to your site and login with the user you want to be an admin
railway run node server/prisma/seed/promoteToOwner.js

And, finally, that's it. All being well, your chapter instance should be up and running.

I know that's a lot, so please let me know if there's anything that needs to be clarified or doesn't work as expected. Hopefully it can be refined in time!

[allella]

Thanks. I'll work on running through these notes and hopefully in the next few days.

[allella]

@ojeytonwilliams I started deploying the Railway template and found it asking for more and more access to GitHub. This seems like too much.

Is all of the following permissions really needed just to deploy an app on Railway?

[allella]

More specifically, why would it need access to All repositories?

If I'm going to limit it, then would I select my fork of Chapter under "Only select repositories" ? If so, do I need to sync up with the upstream before deploying?

[ojeytonwilliams]

Yes, that's correct. Since Railway asks for quite a lot of permissions, I only grant it access to specific repositories - the fork of Chapter in this case. And, yes, it would be a good idea to bring your fork up to date.

[gikf]

Sorry, I didn't see added instructions earlier.

9. Add AUTH0_AUDIENCE, AUTH0_DOMAIN (from dashboard > applications > APIs > Auth0 Management API and dashboard > APIs >Auth0 Management API, respectively)

Second one should be dashboard -> Applications -> settings I think.

---

I've followed instructions, but I'm getting both client and server builds failing.

Before starting builds, I had to manually change Source Repo both for client and server (they were both gikf/chapter Server I believe).

In the build logs, errors start from postInstall.js

Log

#11 83.38 --------------------------
#11 83.38 🎉 WELCOME TO CHAPTER 🎉
#11 83.38 --------------------------
#11 83.38 fatal: not a git repository (or any of the parent directories): .git

#11 83.39 Error: Command failed: git remote -v
#11 83.39 fatal: not a git repository (or any of the parent directories): .git
#11 83.39
#11 83.39     at checkExecSyncError (node:child_process:871:11)
#11 83.39     at execSync (node:child_process:943:15)
#11 83.39     at setup (/app/scripts/postInstall.js:13:18)
#11 83.39     at Object.<anonymous> (/app/scripts/postInstall.js:55:1)

#11 83.39     at Module._compile (node:internal/modules/cjs/loader:1159:14)
#11 83.39     at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
#11 83.39     at Module.load (node:internal/modules/cjs/loader:1037:32)
#11 83.39     at Module._load (node:internal/modules/cjs/loader:878:12)
#11 83.39     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
#11 83.39   status: 128,
#11 83.39   signal: null,
#11 83.39   output: [
#11 83.39     null,
#11 83.39     '',
#11 83.39     'fatal: not a git repository (or any of the parent directories): .git\n'
#11 83.39   ],
#11 83.39   pid: 328,
#11 83.39   stdout: '',
#11 83.39   stderr: 'fatal: not a git repository (or any of the parent directories): .git\n'
#11 83.39 }
#11 83.39 You don't have a .env
#11 83.39 Copying .env.example to .env
#11 83.39 Copied!

Then docker build fails

Log

#13 [stage-0  9/10] RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build
 
#13 0.708 npm
WARN config production Use `--omit=dev` instead.
#13 0.724
#13 0.724 > chapter@0.0.1 build
#13 0.724 > cross-env NODE_ENV=production docker compose -f docker-compose.yml build
#13 0.724
#13 0.816 node:events:491
#13 0.816       throw er; // Unhandled 'error' event
#13 0.816       ^
#13 0.816
 
#13 0.816 Error: spawn docker ENOENT
#13 0.816     at ChildProcess._handle.onexit (node:internal/child_process:283:19)
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
#13 0.816 Emitted 'error' event on ChildProcess instance at:
#13 0.816     at ChildProcess._handle.onexit (node:internal/child_process:289:12)
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
#13 0.816   errno: -2,
#13 0.816   code: 'ENOENT',
#13 0.816   syscall: 'spawn docker',
#13 0.816   path: 'docker',
#13 0.816   spawnargs: [ 'compose', '-f', 'docker-compose.yml', 'build' ]
#13 0.816 }
#13 0.816
#13 0.816 Node.js v18.12.1
 
#13 ERROR: process "/bin/bash -ol pipefail -c npm run build" did not complete successfully: exit code: 1
-----
> [stage-0  9/10] RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build:
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
#13 0.816   errno: -2,
#13 0.816   code: 'ENOENT',
#13 0.816   syscall: 'spawn docker',
#13 0.816   path: 'docker',
 
#13 0.816   spawnargs: [ 'compose', '-f', 'docker-compose.yml', 'build' ]
#13 0.816 }
#13 0.816
#13 0.816 Node.js v18.12.1
-----
 
Dockerfile:25
-------------------
23 |     # build phase
24 |     COPY . /app/.
25 | >>> RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build
26 |
27 |
-------------------
 
ERROR: failed to solve: process "/bin/bash -ol pipefail -c npm run build" did not complete successfully: exit code: 1
 
Error: Docker build failed

[ojeytonwilliams]

Second one should be dashboard -> Applications -> settings I think.

Ooops. I'd describe it as dashboard -> Applications -> your-application -> Settings, but, yes, my instructions were wrong. Thanks, I'll update them!

[ojeytonwilliams]

Okay, so the postInstall error just needs to be suppressed. It's only telling us that it cannot set up an upstream remote and we don't need that for deployment.

As for the rest of the errors, it looks like the template doesn't include the custom build command. Instead it runs npm run build from the root. That's no good for us, because railway doesn't support docker compose (yet, hopefully they add it).

Since the client and server both rely on the /common directory, I can't simply set the root to be /client and /server respectively. Instead, you have to configure those commands separately.

For the client service go to

• Settings > Build > Build Command and set it to npm run -w=client build

Similarly for the server service

• Settings > Build > Build Command and set it to npm run -w=server build

Monorepos are hard to deploy, to say the least. If we can migrate the server to NextJS, this whole thing should get a lot easier, but I'm not sure that's realistic at the moment.

[gikf]

That helped with the builds 🙆‍♂️. Although that either went awry, or database need to be additionally initialized for the server. Adding && npm run db:migrate:reset to the build command helped with it.

Couple notes:

• I'm still a bit confused whether there really isn't way to start initial build & deployment on the railway page, without some shenanigans with removing linked repository and adding it again. After the first build & deployment was attempted, re-deployment is triggered automatically.
• Railway copies client and server addresses without the https:// this will cause errors during build (client) or after deploying (server). So it'd be good to put emphasis on using the full url.
• Auth0 application that I configured for local development worked fine after adding railway urls. Somehow when creating new one something was not working.

[ojeytonwilliams]

Hey @gikf, sorry for the delay getting back to you. I need to take a little time off, but I should be back to 100% next week.

[gikf]

@ojeytonwilliams No worries!