bandolier-website/rendered/docs/long-start.md

{ "author": "Zed A. Shaw", "date": "Dec 30, 2022", "has_image": false, "tag": "Demo", "icon": "rocket", "summary": "A 'long start' guide that makes a single feature that uses the entire stack." }

Long Start Making a Page That Does Everything

You should go through the quick start before doing this tutorial. This one takes you from your newly installed bando setup and teaches you how to create a massively overcomplicated number guessing game that uses every feature of the system. The game is simple, but we'll tour through the whole framework so you know where everything is and what it does.

Experimental Status I'm trying an idea where the longer guide to this project is written as a series of "challenge lessons", and if you can't solve them or they're too vague, you can dive deeper into in-depth HOWTO Recipes on each section. The idea is this might make this guide more interesting and educational. Take a look at this first draft with only the "challenge mode" sections written, and see how far you get. If you enjoy this style, then as I write the in-depth HOWTO Recipes you can try them to learn more. Also let me know if the sections are too vague, or if they just don't really work for you for some reason.

The Goal

This tutorial will create the most incredibly over-engineered number guessing game in the history of the universe. Nearly every feature we'll be adding to the game is overkill, but keeping the goal simple helps us explore every element of the stack without getting stuck also working on the features of the goal.

A number guessing game is simply where we display an input and a prompt to guess a number. Then we tell the user if they get it right or wrong. That's it. We'll then take this "game" to an absurd level by using every feature of the bando that we can. The total time to complete the tutorial is anywhere from a few hours to a few days, depending on your experience level, bugs you might encounter in my code, or mistakes you make.

The Three Core Philosophies

There are three important design philosophies which will help you understand how to work with the bando:

1. Low Friction Progress. You shouldn't have to create and manage tons of files just to get started. You will eventually be juggling as many files as you have features, but in the beginning you'll make one file, and lightly edit one other. As you progress you'll add one or two files, and as you work everything mostly keeps working in "fake" ways while you build up your idea. A lot of other things work without configuration, and if there is configuration it's hopefully clear where to find it. Finally, there's many templates in the "Djenterator" that help you write out a stock file you need to get started, and you can easily add your own templates.
2. UI First Development. When starting a web application it's easier to start with the UI, and then use that to drive the rest of the application. This works better today because the UI tends to change less frequently, is easier to change later, but is more difficult to get right in the beginning. I've found if I spend a lot of time working out the UI, refining it, simplifying it, and determining the data it needs, then all of my backend services fall into place easily. When I go the other direction I'm working in a vacuum that has me crafting useless features the UI won't actually need "just in case."
3. Copying is the Easiest Way to Learn The bando comes with a ton of components not so you can use them blindly, but more so you can extend them, or outright replace them later. You can just grab the stock Video.svelte to get your video up, work on your UI, and add what you need to Video.svelte. If you keep adding hacks to Video.svelte and it's breaking your brain then that's a prime time to rewrite it. This helps you create your vision, but also learn how all of these components and features are created so you're not dependent on anyone. Everyone learns by copying others, and the bando promotes copying and rewriting as a first class principle by not hiding any of the code from you.

These ideas are different from other frameworks you'll encounter, so you'll have to adapt later when you start picking up other technologies. For example, many web frameworks require you to build almost everything from scratch. Need to show some videos? Time to build your own HLS video plugins. Need authentication? Time to reinvent bcrypt password storage all over again. Other frameworks seem to idolize building everything from scratch...except the things they've written already. If you try to rebuild that they say you have "Not Invented Here Syndrome." "Reinventing the Wheel." "Don't look at the man behind the curtain."

Other frameworks also seem to be enamored with their byzantine configuration files and require you to lay down 4, 5, or sometimes even 10 or more additional files just to get started. Want to just get that first page up? Tough. You'll need the page, a form, a view, a route, a class in your route, a model, the controller, oh did I mention the controller is also called a view on some places? You also need to mirror a 3-4 levels deep directory structure for...reasons. Even worse is how other frameworks push the idea that all this complexity somehow makes them more "professional" or "grown up" when the truth is it's just more complex, not more professional. PHP doesn't have all this BS, and PHP rakes in trillions of dollars a year at the biggest companies we've ever seen.

The bando is meant to be educational, so it tries not to overload you with everything all at once. My goal is to gradually introduce each piece so you can focus on learning each piece of the stack. Then when you've learned how most of these things work you should be able to jump over to other frameworks that throw a firehose of information at you right away, or that make you build your own authentication yet again.

The Flat Directory Structure

You should familiarize yourself with the directory structure described in the Quick Start before you continue. Here's what each directory contains:

• admin - This is where the admin control panel lives.
• api - The JSON api handlers live here.
• bando.js - This is your main management script.
• bando.ps1 - This is a Windows compatible version of the script.
• build.json - A build configuration used by the commands/build.js command that runs esbuild.
• build.prod.json - The production build configuration used in the npm run build command.
• client - This is where the main web application lives, and is a dynamic Svelte front-end.
• commands - The bando.js script runs commands out of here.
• coverage - You won't see this at the start, but if you run the coverage commands then you'll see code coverage output here.
• debug - Various debugging outputs end up here.
• dev.sqlite3 - SQLite3 is the default database (PostgreSQL coming soon).
• dev.sqlite3-shm - You'll see this because the SQLite3 database is configured for performance.
• dev.sqlite3-wal - Same as above.
• emails - Your email templates and configurations are in here. Edit these to change how you email your users.
• knexfile.cjs - This is the database configuration using knex.js.
• lib - Various support utilities for are found in here, but are only for non-browser tools. Look in client/ for admin and client imports.
• media - If you do videos or audio then this can be a separate directory for media. You need this so you can wipe public/ at any time.
• migrations - The knex.js migrations. You'll see all that I've made over development so you have many examples.
• node_modules - Your modules when you run npm install.
• nodemon.json - The package.json uses nodemon to trigger builds when you change something. Esbuild handles this for admin and client code though.
• package-lock.json - npm makes this, and you can search in here to see exact versions of your packages and what depends on what.
• package.json - This configures the project. Take special node of the "type": "module" configuration, which configures node for ES6 style ESM imports.
• public - This directory should be safe to empty if you need to debug how things are being built. It is constructed from all of the other directories to create the content you would place on your webserver.
• queues - This directory has queue handlers, which use Bull to offload long running processing. Email sends, Discord bot handling, Paypal notifications, Stripe notifications, and Livestream notifications are handled by these.
• rendered - This is where the rendered static pages go, and you can look in rendered/pages/blog to see an example of using this.
• scripts - These are mostly scripts and "junk" other modules need, like Svelte's TypeScript configuration script, or PM2 configuration examples.
• secrets - NEVER PUT THIS IN GIT. This is where your secret configuration files go. This will contain Payment keys, Discord keys, and other special configurations you don't want people to get.
• socket - This contains the socket.io handlers that give easy asynchronous communication with the browser.
• static - These are static files that need to be copied over to public/. It's things like icons, images, browser JavaScript code, etc.
• tests - Contains the automated ava tests for the application. The majority of them use Playwright to run the browser like a user to confirm things are still working.

It may seem like there's a lot of directories but that's only because it's a flat structure with very limited nesting. Other frameworks have nearly the same number, they just hide them in deep hierarchies that are difficult to remember and search. Once you have a grip on this then continue with the lesson.

HOWTO Recipes

This tutorial is structured in a very terse quick language meant to get you going, and if nothing bad happens, get something up quickly. The problem is, well, bad things always happen in computers, so that's why each section below will link to larger explanations in the HOWTO Recipes section of the site. If you run into trouble in a section, or you want to diver deeper into that feature of the framework, then use the links at the end of the section to explore more.

Step 1: The Unstyled Fake UI

You should have finished the Quick Start so you have a working bando starter going. You also created a first test page called client/pages/Test.svelte and added it to the client/routes.js file. You should now undo that before continuing with a new page, just to make sure you know how it works:

1. Remove Test.svelte from the client/routes.js.
2. Delete the client/pages/Test.svelte file.
3. ctrl-c the app and restart it with npm run DANGER_ADMIN just to make sure everything is ready.

Once you're sure everything is working you'll make a new page for the number guessing:

./bando.js djent --template ./static/djenterator/client.svelte --output client/pages/NumberGuess.svelte`.

Then add it to the client/routes.js just like you did with Test.svelte. I'd put the page at /number_guess/ to start, so in your browser you should be going to http://127.0.0.1:5001/client/#/number/ to see the initial message.

Once it's loading you'll change the api.mock() to look like this:

api.mock({
  "/api/number_guess": {
    "get": [200, {"number": 340, "guesses": 10}],
  }
});

Then update the onMount to have this /api/number_guess URL:

const [status, data] = await api.get("/api/number_guess");

This will pretend to work, and later you'll delete this api.mock(), but for now it lets you develop the UI without more gear.

Read the HOWTO Recipe for More

Step 2: The Blockstart Fake UI

Next, you'll import the client/components/Blockstart.svelte component so you can do a quick layout. After watching many designers work I've found that they almost universally start with a big blocky layout, or they dump all the media onto a page and then organize it with a big blocky layout.

The Blockstart.svelte gives you quick access to some prototyping layout tools with a few simple tags, so you can work entirely in HTML to get your initial design working.

Once you have your basic layout you'll create a form for the user to input their guess and then submit it. You can work at this stage until your fake page is doing a simple guess loop.

To use Blockstart.svelte you wrap what you want to layout with a <Blockstart> tag and then use the blockstart tags to get your initial layout done.

Read the HOWTO Recipe for More

Step 3: The Styled Fake UI

Now that you have your Fake UI mostly laid out it's time to "pull out" the Blockstart.svelte and convert it to CSS rules. The best way to do this is this:

1. Create your <style></style> block in your NumberGuess.svelte.
2. Pick a tag to convert, and give it a name, then add that name to <style> as a rule.
3. Go through each tag you've written with block start and pull out the style="" configurations and copy them to your tag's rule.
4. Add the required display: flex and flex-direction: for your tag to get it back where it was. Remember hoROWzontal and vertiCOLUMN if you can't remember how flex-direction: row and flex-direction: column are oriented.
5. Repeat this process for each tag, slowly building your own CSS until you don't need the <Blockstart> anymore, then delete it.

Read the HOWTO Recipe for More

Step 4: Testing The Fake UI

Next you want to create a simple test that loads the page to make sure it keeps working. You'll start by using bando.js djent to generate a first test:

node bando.js djent --template ./static/djenterator/ui_test.js --output tests/ui/number_guess.js

Edit the tests/uit/number_guess.js file and change the test.before to this:

test.before(async t => t.context = await playstart('http://127.0.0.1:5001/client/#/number_guess/'));

Then delete the line that attempts a registration:

const user = await register_user();

After that you'll use the lib/testing.js library, and the Playwright API to write your test. Refer to other tests in the test/ui/ directory for examples of many testing scenarios.

Read the HOWTO Recipe for More

Step 5: The Fake api/ Handler

As mentioned in the intro to this document we're extremely over engineering this. I don't think there's any reason to create a JSON API just to do a number guessing game in real life, but then again, you wouldn't be that serious about a number guessing game to begin with.

If you've been developing your Fake UI and UI Test you should have a fake data with the client/api.js that probably looks like this:

api.mock({
  "/api/number_guess": {
    "get": [200, {"number": 340, "guesses": 10}],
  }
});

This is in your NumberGuess.svelte and we now need to move this into an api/ handler. First, use bando.js djent to make a basic handler for you:

node bando.js djent --template ./static/djenterator/api.js --output api/number_guess.js

Edit the api/number_guess.js file and change the reply_data variable in get to simply be what your current fake data is returning:

export const get = async (req, res) => {
  const api = new API(req, res);
  const reply_data = {"number": 340, "guesses": 10};

Then you delete the api.mock from the NumberGuess.svelte and assuming everything is done right your page should keep working, but now you'll see messages logged in the terminal from the api/number_guess.js handler like this:

[api           ] DEBUG [1672738061074] (84203 on mycomputer): DEBUG[84203] api/number_guess.js:  {"number":340,"guesses":10}

Now you'll work on the backend of the service by writing the code to create guesses, check guesses, validate input, and return results. No database yet so just attach the guesses to the api.user (req.user) until the next part.

Read the HOWTO Recipe for More

Step 6: The Database Table

You'll want to store the user's current game in the database, so you need to create a knex.js migration, then edit the migration to create the table for your game. You first create the migration with:

npm run knex migrate:make number_guess_table

This will create a file named something like migrations/20230103095413_number_guess_table.cjs but with a different number (based on the date). Then you can generate a better starter file with:

node bando.js djent --template ./static/djenterator/migration.js --output migrations/20230103095413_number_guess_table.cjs --force

The --force option will overwrite the original file knex.js created. Now you can edit it to create your table like this:

exports.up = async (knex) => {
  await knex.schema.createTable('number_guess', (table) => {
    table.increments('id');
    table.timestamps(true, true);

    table.integer('player_id').notNullable();
    table.foreign('player_id').references('id').inTable('user');

    table.integer('number').notNullable().default(0);
    table.integer('guesses').notNullable().default(0);
  });
};

exports.down = async (knex) => {
  await knex.schema.dropTable('number_guess');
};

You can then create this table with knex:

npm run knex migrate:latest

If you make a mistake or want to change the table's schema, it's best to rollback and then run the migration again:

npm run knex migrate:down
npm run knex migrate:latest

Finally, restart the server completely so the new database table is picked up in the SCHEMA.

Read the HOWTO Recipe for More

Step 7: lib/models.js and Model Testing

You then want to add this one line to lib/models.js to gain access to your new number guessing database table:

export class NumberGuess extends Model.from_table('number_guess') {
}

That's all you need, and now you can go to http://127.0.0.1:5001/admin/#/table/number_guess/ and see your empty table. Use the + icon to create a new record assigned to player_id=1 which should be your admin user while you're testing.

You can happily use the database admin to test out your handler while you work on it, but you should probably spend a bit of time to write a tests/models test for your new NumberGuess model. Look at the other files for examples.

You'll need the ormish apis to work with the NumberGuess model. You'll want to have your test do the basic Create-Read-Update-Delete operations, and that should be good to start.

Read the HOWTO Recipe for More

Step 8: The api/ to the Database

You now have everything you need to make the final connection:

1. Your NumberGuess.svelte can talk to the /api/number_guess for number guessing.
2. You have a NumberGuess model in lib/models.js.
3. Your api/number_guess.js can now work with the NumberGuess model to create a fully working number guessing game for the gods!

You'll need the ormish apis to work with the NumberGuess model. If you were smart and wrote a tests/models/number_guess.js then you should have most of what you need to make this work in your api/number_guess.js file.

Read the HOWTO Recipe for More

Step 9: Adding Registration and Authentication

You've already seen the authentication system in action when you first installed the application. You can go look at your User table in the admin tool and search for your email. The search is very simple but should work for your basic setup. You'll notice that the password field is encrypted, and that uses the bcryptjs library which is a pure JavaScript version of bcrypt.

I use bcryptjs because the C/C++ dependencies for bcrypt frequently break on almost every platform. I ran into so many issues reliably installing the bcrypt library that I just switched to the JavaScript one for most of the development. I find that for small to medium sites it's not a big problem, but if you get big then make the switch in production. The APIs are supposed to be the same so the switch should be simply importing the right modules.

To enable authentication on your NumberGuess.svelte you need to add authentication to the <Layout> tag:

<Layout centered={ true } authenticated={ true }>

You can also set auth_optional={true} instead if you want people to view the game, but then register/authenticate optionally. For now just set it to authenticated.

You're not done yet! Setting this page to authenticated doesn't actually make the game authenticated. You also need to tag your api/number_guess.js handler to be authenticated like this:

get.authenticated = true;

That will require authentication before it will do any operations. Without this your visual UI will pretend to be authenticated, but then anyone with a CURL command can just do whatever they want.

Finally, you'll need to update your test to bring back the registration line you removed so that you register a user before going to your page, but now we want to put the user variable at the top so we can delete it in the teardown:

import { User } from "../../lib/models.js";

let user;

test.before(async t => t.context = await playstart('http://127.0.0.1:5001/client/#/number_guess/'));

test.after(async t => {
  await playstop(t.context.browser, t.context.p);
  if(user) await User.destroy(user);
});

test('test /test_me works', async (t) => {
  const {browser, context, p} = t.context;
  user = await register_user();

The sticky part here will be registration. For now you can leave in the stock registration but if you were to get serious about this very serious number guessing game then you'd want to juggle a more delayed approach. We'll work on that later.

Read the HOWTO Recipe for More

Step 10: Static Leaderboard Command

The bando features a simple "command" system that you can enhance with your own commands. This promotes automating your work by making it easy to craft automation tooling in the commands/ directory. You've been using this automation when you run node bando.js djent, so now you'll write your own command.

First you'll use bando.js djent to create a stock command:

node bando.js djent --template ./static/djenterator/command.js --output commands/leaderboard.js

When you run node bando.js help you should see your fresh command listed like this:

 leaderboard [options] <source>  Describe your command here.

You can then run this command to test it out with:

node bando.js leaderboard
node bando.js leaderboard --output test.txt
node bando.js leaderboard --output test.txt output.txt

The first two runs of leaderboard should produce errors and the final one some simple debug output. This project uses the commander module to process CLI options, but normalizes it to just setting some variables. Take the time now to read the code to bando.js to see that it's actually not that complex. Mostly just loads all the modules in commands/ and sets them up with commander.

Edit the commands/leaderboard.js file and start filling in the code for generating the leaderboard statistics for your game. My idea for this step is you'll have to:

1. Change the number_guess table to have fields for wins and losses.
2. The commands/leaderboard.js will then use lib/models.js:NumberGuess to query this and generate a a public/leaderboard.json file.
3. You can then create a UI that loads this /leaderboard.json file to display the site's current winners and losers.
4. Once your commands/leaderboard.js is working, you can place it in a cronjob to run every 5 minutes to update the /leaderboard.json file.

This little task will involve revisiting many of the previous features as you have to create:

1. A new migration for the number_guess table.
2. A new Svelte UI for the new /leaderboard.json.
3. More testing in tests/models/number_guess.js.
4. The new commands/leaderboard.js to generate the public/leaderboard.json file.

Read the HOWTO Recipe for More

Step 11: Emailing Winners

Once you have a Leaderboard working you can start to email the winners. You can use the lib/emails.js module to send emails, and you can create templates in emails/. You'll need:

1. emails/winners.txt -- This is the text version of the email.
2. emails/winners.html -- This is the HTML version. You can try to use one of the example templates, but I suggest simply creating a lightly "HTMLified" version of your text. If you're smarter than me you'd just have your winners.txt automatically convert to winners.html with the built-in Markdown in lib/docgen.js.
3. The main function to use in lib/email.js is send_email.
4. You'd then modify your commands/leaderboard.js to setup the email and send it.

While you're developing the emails you'll want to run everything with DEBUG=1 set like this:

DEBUG=1 npm run DANGER_ADMIN

On Windows do it like this:

$env:DEBUG=1
npm run DANGER_ADMIN

Then when you run your tests do the same thing:

DEBUG=1 npm run test

This will configure the email to write the rendering of emails to the debug/emails directory so you can open them in your editor or browser. This is necessary because testing emails using real sending is incredibly difficult.

Read the HOWTO Recipe for More

Step 12: Sending Emails with Queues

Once you have emails working directly you can create a queue/ handler to send it instead. In a script like commands/leaderboard.js this isn't very necessary, but you would use a queue for sending emails inside your api/ handlers. The reason you need a queue is so your handling of web requests isn't blocked by email server processing.

For education though we're just going to modify commands/leaderboard.js to do it. Here's what you'd need to do:

1. Open the queues/mail.js file to add your own handler.
2. Grab a simple function that's there already like welcome and copy it.
3. Use the load_templates function to create your own variable similar to welcome_form, but load_templates("winners").
4. Edit lib/queues.js to quickly add a little send_winners function similar to send_welcome.

Read the HOWTO Recipe for More

Step 13: Advanced Fancy FSM Based UI

Clearly your number guessing game is not complex enough. We need to add some advanced math to make it really impress all the Haskell programmers out there. To do that we're going to redesign your number guessing logic using the client/fsm.js library.

You should copy your current UI over to one name client/pages/FSMNumberGuess.svelte and then edit the client/routes.js to point at this instead. Your tests should keep working as all you've done is change the implementation at the same URL.

Now you need to import the FSM and use the client/fsm.js docs to create a GuessEvents class to handle the user's clicks and inputs.

Coming soon. I have to actually write this UI and then also convert it to an FSM to get these instructions right.

Read the HOWTO Recipe for More

Step 14: Delayed Optional Registration

You're currently requiring a registration to play, but that won't do if you want to receive VC investment for your over engineered number guess game. You have an FSM for your UI, so you can create a more complex UI that lets people play, but make them register to keep their score and share it on the Leaderboard. The best way to do this is have your FSM allow play, and then after a few guesses pop up a client/components/Toasts.svelte saying they can register. The FSM is really designed for this kind of complex state management.

Another option is to get them to play a game, and then when they win ask if they want to register to save the score.

One more suggestion is to constantly show them people beating the game in notifications using the next feature you make.

Read the HOWTO Recipe for More

Step 15: WebSocket Winner Notification

The final piece of the stack to play with is the socket/ handlers. As this is insanely over engineered I think the easiest thing to implement is notifying people that someone else just beat the game. If you look in the user table you'll notice the following fields:

• initials -- When people register they can give a few letters to display publicly.
• full_name -- This is their internal name that you probably shouldn't show random people. I use this mostly for finding people's purchases, emailing them privately, etc.
• email -- This is also an internal field, which you shouldn't show to anyone else.

That means, you can probably do this:

1. When someone registers you can start tracking their wins and losses.
2. When they win, you can notify a queue that they won.
3. That queue handler can then do the work of sending an announcement to the socket/ so that it tells everyone currently playing that this person won.
4. Here's the catch, you should just tell everyone the initials of the winner, and maybe their current score on the Leaderboard. You should not tell anyone else the other information in their user record.

That's a decent chunk of work, and you should already know the queue/ system, so you need to learn the socket/ system. Here's what you need to learn:

1. Writing a handler in socket/, take a look at socket/chat.js.
2. Using lib/websocket.js to talk to the backend handler in socket/.
3. Connecting it to a queue/ so that your web application can fire it off and keep running.

Once you can get this working you have created something in every feature of The Bandolier.

Read the HOWTO Recipe for More