Browse Source

An absolute bunch of trash but it works. I now have to figure out how to rip out gulp and just write a better generator.

pull/20/head
Zed A. Shaw 2 months ago
parent
commit
ad68cbd2a0
45 changed files with 1705 additions and 235 deletions
  1. +2
    -1
      .gitignore
  2. +7
    -0
      README.md
  3. +0
    -111
      SAPPER.md
  4. +7
    -0
      db/blog/index.json
  5. +14
    -0
      db/blog/posts/01-first-blog-post.md
  6. +14
    -0
      db/games/0.json
  7. +14
    -0
      db/games/1.json
  8. +0
    -0
      db/live/status.json
  9. +0
    -0
      db/live/stream.json
  10. +21
    -0
      db/modules/drawing_basics/exercises/01-sight-sized.md
  11. +11
    -0
      db/modules/drawing_basics/index.json
  12. +6
    -1
      db/modules/html_basics/exercises/01-intro.md
  13. +11
    -0
      db/modules/html_basics/index.json
  14. +23
    -0
      db/modules/index.json
  15. +12
    -0
      db/protos/videos.json
  16. +10
    -0
      db/slides/1.json
  17. +12
    -0
      db/videos/1.json
  18. +108
    -31
      gulpfile.js
  19. +0
    -1
      media/blog/index.json
  20. +14
    -0
      media/blog/posts/01-first-blog-post.md
  21. +0
    -1
      media/modules/0-html/index.json
  22. +21
    -0
      media/modules/drawing_basics/exercises/01-sight-sized.md
  23. +18
    -9
      media/modules/html_basics/exercises/01-intro.md
  24. +45
    -0
      notes/video_encode.md
  25. +992
    -1
      package-lock.json
  26. +10
    -0
      package.json
  27. +1
    -1
      src/node_modules/docs.js
  28. +3
    -0
      src/routes/blog/[slug].svelte
  29. +4
    -4
      src/routes/blog/index.svelte
  30. +7
    -7
      src/routes/modules/[slug]/[exercise].svelte
  31. +15
    -28
      src/routes/modules/[slug]/index.svelte
  32. +19
    -17
      src/routes/modules/index.svelte
  33. +1
    -0
      static/api/blog/index.json
  34. +1
    -0
      static/api/blog/posts/01-first-blog-post.json
  35. +1
    -0
      static/api/modules/drawing_basics/exercises/01-sight-sized.json
  36. +1
    -0
      static/api/modules/drawing_basics/index.json
  37. +1
    -0
      static/api/modules/html_basics/exercises/01-intro.json
  38. +1
    -0
      static/api/modules/html_basics/index.json
  39. +23
    -0
      static/api/modules/index.json
  40. +11
    -0
      tools/convert.sh
  41. +11
    -0
      tools/convert_webm.sh
  42. +140
    -22
      tools/gulp.js
  43. +3
    -0
      tools/sync.sh
  44. +42
    -0
      tools/torrents.prod.sh
  45. +48
    -0
      tools/torrents.sh

+ 2
- 1
.gitignore View File

@@ -106,7 +106,6 @@ yarn-error.log
yarn-error.log
/cypress/screenshots/
/__sapper__/
/db/

.sessions
static/streams
@@ -114,4 +113,6 @@ dump.rdb
devices/
lib/secrets.js
*.mp4
*.webm
*.jpg
secret/

+ 7
- 0
README.md View File

@@ -57,3 +57,10 @@ Wed Mar 4 16:37:50 EST 2020
Quite a lot of the design is refined and the development process is much smoother.
Close to throwing it on the internet and seeing how it goes.


## Important Docs

These are links to docs I constantly have to look up:

* https://github.com/gulpjs/vinyl/blob/master/README.md


+ 0
- 111
SAPPER.md View File

@@ -1,111 +0,0 @@
# sapper-template

The default [Sapper](https://github.com/sveltejs/sapper) template, available for Rollup and webpack.


## Getting started


### Using `degit`

[`degit`](https://github.com/Rich-Harris/degit) is a scaffolding tool that lets you create a directory from a branch in a repository. Use either the `rollup` or `webpack` branch in `sapper-template`:

```bash
# for Rollup
npx degit "sveltejs/sapper-template#rollup" my-app
# for webpack
npx degit "sveltejs/sapper-template#webpack" my-app
```


### Using GitHub templates

Alternatively, you can use GitHub's template feature with the [sapper-template-rollup](https://github.com/sveltejs/sapper-template-rollup) or [sapper-template-webpack](https://github.com/sveltejs/sapper-template-webpack) repositories.


### Running the project

However you get the code, you can install dependencies and run the project in development mode with:

```bash
cd my-app
npm install # or yarn
npm run dev
```

Open up [localhost:3000](http://localhost:3000) and start clicking around.

Consult [sapper.svelte.dev](https://sapper.svelte.dev) for help getting started.


## Structure

Sapper expects to find two directories in the root of your project — `src` and `static`.


### src

The [src](src) directory contains the entry points for your app — `client.js`, `server.js` and (optionally) a `service-worker.js` — along with a `template.html` file and a `routes` directory.


#### src/routes

This is the heart of your Sapper app. There are two kinds of routes — *pages*, and *server routes*.

**Pages** are Svelte components written in `.svelte` files. When a user first visits the application, they will be served a server-rendered version of the route in question, plus some JavaScript that 'hydrates' the page and initialises a client-side router. From that point forward, navigating to other pages is handled entirely on the client for a fast, app-like feel. (Sapper will preload and cache the code for these subsequent pages, so that navigation is instantaneous.)

**Server routes** are modules written in `.js` files, that export functions corresponding to HTTP methods. Each function receives Express `request` and `response` objects as arguments, plus a `next` function. This is useful for creating a JSON API, for example.

There are three simple rules for naming the files that define your routes:

* A file called `src/routes/about.svelte` corresponds to the `/about` route. A file called `src/routes/blog/[slug].svelte` corresponds to the `/blog/:slug` route, in which case `params.slug` is available to the route
* The file `src/routes/index.svelte` (or `src/routes/index.js`) corresponds to the root of your app. `src/routes/about/index.svelte` is treated the same as `src/routes/about.svelte`.
* Files and directories with a leading underscore do *not* create routes. This allows you to colocate helper modules and components with the routes that depend on them — for example you could have a file called `src/routes/_helpers/datetime.js` and it would *not* create a `/_helpers/datetime` route


### static

The [static](static) directory contains any static assets that should be available. These are served using [sirv](https://github.com/lukeed/sirv).

In your [service-worker.js](src/service-worker.js) file, you can import these as `files` from the generated manifest...

```js
import { files } from '@sapper/service-worker';
```

...so that you can cache them (though you can choose not to, for example if you don't want to cache very large files).


## Bundler config

Sapper uses Rollup or webpack to provide code-splitting and dynamic imports, as well as compiling your Svelte components. With webpack, it also provides hot module reloading. As long as you don't do anything daft, you can edit the configuration files to add whatever plugins you'd like.


## Production mode and deployment

To start a production version of your app, run `npm run build && npm start`. This will disable live reloading, and activate the appropriate bundler plugins.

You can deploy your application to any environment that supports Node 10 or above. As an example, to deploy to [ZEIT Now](https://zeit.co/now) when using `sapper export`, run these commands:

```bash
npm install -g now
now
```

If your app can't be exported to a static site, you can use the [now-sapper](https://github.com/thgh/now-sapper) builder. You can find instructions on how to do so in its [README](https://github.com/thgh/now-sapper#basic-usage).


## Using external components

When using Svelte components installed from npm, such as [@sveltejs/svelte-virtual-list](https://github.com/sveltejs/svelte-virtual-list), Svelte needs the original component source (rather than any precompiled JavaScript that ships with the component). This allows the component to be rendered server-side, and also keeps your client-side app smaller.

Because of that, it's essential that the bundler doesn't treat the package as an *external dependency*. You can either modify the `external` option under `server` in [rollup.config.js](rollup.config.js) or the `externals` option in [webpack.config.js](webpack.config.js), or simply install the package to `devDependencies` rather than `dependencies`, which will cause it to get bundled (and therefore compiled) with your app:

```bash
npm install -D @sveltejs/svelte-virtual-list
```


## Bugs and feedback

Sapper is in early development, and may have the odd rough edge here and there. Please be vocal over on the [Sapper issue tracker](https://github.com/sveltejs/sapper/issues).

+ 7
- 0
db/blog/index.json View File

@@ -0,0 +1,7 @@
{
"author":"Zed A. Shaw",
"url":"/blog/",
"title": "Learn JS The Hard Way",
"subtitle": "A blog about teaching JS on the internet.",
"posts": []
}

+ 14
- 0
db/blog/posts/01-first-blog-post.md View File

@@ -0,0 +1,14 @@
{
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"has_image": true,
"summary": "A new kind of blog post for a new kind of blog.",
"title": "A First Real Blog Post"
}

------
# The First Real Blog Post

I've been working on this new system for publishing and managing my courses for about a month. It's actually been in my mind for about two years, but only recently all the pieces of JavaScript technology clicked together to make it easily possible. In this first blog post I'm going to go through the features and talk about why I'm working on this live.



+ 14
- 0
db/games/0.json View File

@@ -0,0 +1,14 @@
{
"id": 0,
"uuid": "",
"title": "Cave Game",
"short_description": "This is a game a made out of a book for PICO-8",
"tags": "#fun #games #tags",
"cover": "/images/400/Cave_Game.jpg",
"genre": "flappy bird",
"description": "This is a game a made out of a book for PICO-8",
"related": [
{"url": "/words/1.json", "type": "words"}
],
"frame_url": "/games/1.html"
}

+ 14
- 0
db/games/1.json View File

@@ -0,0 +1,14 @@
{
"id": 1,
"uuid": "",
"title": "Lander Game",
"short_description": "A little lunar lander PICO-8 game.",
"tags": "#fun #games #tags",
"cover": "/images/400/Cave_Game.jpg",
"genre": "flappy bird",
"description": "This is a game a made out of a book for PICO-8",
"related": [
{"url": "/words/1.json", "type": "words"}
],
"frame_url": "/games/2.html"
}

+ 0
- 0
db/live/status.json View File


+ 0
- 0
db/live/stream.json View File


+ 21
- 0
db/modules/drawing_basics/exercises/01-sight-sized.md View File

@@ -0,0 +1,21 @@
{
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"module": "/modules/drawin_basics",
"id": 1,
"title": "Site Sized Drawing",
"completed": false,
"image": "/images/hero-holder.svg",
"summary": "The first lesson where you learn about basic drawing terms and how to do sight sized drawing.",
"video": {
"src": "/videos/drawing_basics/Drawing_Basics_01_Sight_Sized.webm",
"poster": "/thumbs/video_stills/400/drawing_basics/01_Sight_Sized.jpg",
"preload": "none"
}
}

------

# Sight Sized Drawing

This is the first exercise where you have some stuff.

+ 11
- 0
db/modules/drawing_basics/index.json View File

@@ -0,0 +1,11 @@
{
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/drawing_basics",
"slug": "drawing_basics",
"title": "Drawing Basics",
"tag": "what",
"subtitle": "Learn just enough drawing to be deadly.",
"summary": "An introduction to drawing with pencil and digitally to get an understanding of it for the web.",
"exercises": []
}

media/modules/0-html/intro.md → db/modules/html_basics/exercises/01-intro.md View File

@@ -1,7 +1,12 @@
{
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"module": "/modules/0-html",
"module": "/modules/html_basics",
"id": 2,
"title": "Basic Terminology",
"completed": false,
"image": "/images/hero-holder.svg",
"summary": "This will first introduce some basic terminology for HTML.",
"video": {
"src": "/images/sample.mp4",
"poster": "/images/sample.jpg",

+ 11
- 0
db/modules/html_basics/index.json View File

@@ -0,0 +1,11 @@
{
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/html_basics",
"slug": "html_basics",
"title": "HTML Basics",
"tag": "what",
"subtitle": "Start here if you know nothing.",
"summary": "An introduction to basic HTML for the web.",
"exercises": []
}

+ 23
- 0
db/modules/index.json View File

@@ -0,0 +1,23 @@
[ {
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/drawing_basics",
"slug": "drawing_basics",
"title": "Drawing Basics",
"tag": "what",
"subtitle": "Learn just enough drawing to be deadly.",
"summary": "An introduction to drawing with pencil and digitally to get an understanding of it for the web.",
"exercises": []
},
{
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/html_basics",
"slug": "html_basics",
"title": "HTML Basics",
"tag": "what",
"subtitle": "Start here if you know nothing.",
"summary": "An introduction to basic HTML for the web.",
"exercises": []
}
]

+ 12
- 0
db/protos/videos.json View File

@@ -0,0 +1,12 @@
{
"id": ID,
"torrent_url": "/torrents/XXX.torrent",
"title": "TITLE",
"short_description": "DESC",
"tags": "#art #pleinair #painting #oilpainting",
"published_on": "2019",
"covers": {"small":"/media/video_stills/thumbs/128/XXX.jpg","medium":"/media/video_stills/thumbs/400/XXX.jpg","large":"/media/video_stills/thumbs/1024/XXX.jpg"},
"quality": "1080p",
"url": "/media/videos/XXX.mp4"
}


+ 10
- 0
db/slides/1.json View File

@@ -0,0 +1,10 @@
{
"id": 1,
"title": "A Test Slides",
"description": "This is just an idea for slides for the site",
"pages": [
"test1/SVG_Test_1.svg",
"test1/SVG_Test_2.svg"
]

}

+ 12
- 0
db/videos/1.json View File

@@ -0,0 +1,12 @@
{
"id": 10,
"torrent_url": "/torrents/drawing_basics_01_sight_sized.torrent",
"title": "Drawing Basics #1 -- Sight Sized",
"short_description": "In this video I show you the basics of sight sized drawing and the basic terminology for the rest of the course.",
"tags": "#art #drawing",
"published_on": "2020",
"covers": {"small":"/media/video_stills/thumbs/128/drawing_basics/01_sight_sized.jpg","medium":"/media/video_stills/thumbs/400/drawing_basics/01_sight_sized.jpg","large":"/media/video_stills/thumbs/1024/drawing_basics/01_sight_sized.jpg"},
"quality": "2160p",
"url": "/media/videos/drawing_basics/01_sight_sized.mp4"
}


+ 108
- 31
gulpfile.js View File

@@ -1,43 +1,120 @@
let gulp = require('gulp');
let sass = require('gulp-sass');
let cleancss = require('gulp-clean-css');
let csscomb = require('gulp-csscomb');
let rename = require('gulp-rename');
let autoprefixer = require('gulp-autoprefixer');
let { split, mdindex, log } = require('./tools/gulp');
const db = 'media';
const gulp = require('gulp');
const autoprefixer = require('gulp-autoprefixer');
const cleancss = require('gulp-clean-css');
const cmd = require('gulp-exec');
const csscomb = require('gulp-csscomb');
const debug = require('gulp-debug');
const del = require('del');
const exec = require('child_process').execSync;
const fs = require('fs');
const gitignore = require('gulp-gitignore');
const notify = require('gulp-notify');
const rename = require('gulp-rename');
const sass = require('gulp-sass');
const glob = require('glob');
const { covers, jsonIndex, markdownJSON, thumbsup, exifJSON, ffprobeJSON} = require('./tools/gulp.js');

let paths = {
source: './sass/*.scss',
};

const media_index = async (cb) => {
try {
await mdindex('./media', './media/blog/**/*.md', './media/blog/index.json');
await mdindex('./media', './media/modules/0-html/*.md', './media/modules/0-html/index.json')
} catch (error) {
console.log(error);
} finally {
cb();
}
}
const media = 'media';
const api = 'static/api';
const db = 'db';

const clean = () => del([`${media}/**/*`, `${api}/**/*`]);

const styles = () => {
return gulp.src(paths.source)
.pipe(sass({outputStyle: 'compact', precision: 10})
.on('error', sass.logError)
)
.pipe(autoprefixer())
.pipe(csscomb())
.pipe(cleancss())
.pipe(rename({
suffix: '.min'
}))
.pipe(gulp.dest('./static/css/'));
.pipe(sass({outputStyle: 'compact', precision: 10})
.on('error', sass.logError)
)
.pipe(autoprefixer())
.pipe(csscomb())
.pipe(cleancss())
.pipe(rename({
suffix: '.min'
}))
.pipe(gulp.dest('./static/css/'));
};

const all = gulp.parallel(styles, media_index);
const modules_headers = async () => {
// if you do it this way each one chains onto the next one so order is important
return gulp.src(`${db}/modules/**/**/*.md`)
.pipe(gulp.dest('media/modules'))
.pipe(markdownJSON('/modules'))
.pipe(rename(path => path.extname = '.json'))
.pipe(gulp.dest(`${api}/modules/`))
}

const gen_indexer = (type_dir, base_dir, merge_target) => {
return async() => {
// load the main index from the db
let index_json = JSON.parse(fs.readFileSync(`${db}/${type_dir}/${base_dir}/index.json`));
// but use the generated api json files to update this index

return gulp.src(`${api}/${type_dir}/${base_dir}/exercises/*.json`)
.pipe(jsonIndex(
base=`/api/${type_dir}/${base_dir}/`,
index=`${api}/${type_dir}/${base_dir}/index.json`,
merge=index_json,
merge_target=merge_target));
}
}
const html_basics = gen_indexer('modules', 'html_basics', 'exercises');
const drawing_basics = gen_indexer('modules', 'drawing_basics', 'exercises');

const modules = gulp.series(modules_headers, html_basics, drawing_basics);

exports.default = () => {
return gulp.watch(['./**/*.scss', './media/**/*.md'], all);
const blog_headers = async () => {
// if you do it this way each one chains onto the next one so order is important
return gulp.src(`${db}/blog/**/*.md`)
.pipe(gulp.dest('media/blog'))
.pipe(markdownJSON('/blog'))
.pipe(rename(path => path.extname = '.json'))
.pipe(gulp.dest(`${api}/blog/`))
}

const blog_index = async () => {
let index_json = JSON.parse(fs.readFileSync('db/blog/index.json'));

return gulp.src(`${api}/blog/posts/*.json`)
.pipe(jsonIndex(
base='/api/blog/',
index=`${api}/blog/index.json`,
merge=index_json,
merge_target="posts"));
}

const blog = gulp.series(blog_headers, blog_index);


const structure = async () => {
return gulp.src([`${db}/**/*.json`, `${db}/**/*.md`], {read: false})
.pipe(gulp.dest(`${api}`)).pipe(gulp.dest(`${media}`));
}

const video_thumbs = async () => {
let input = gulp.src(`${media}/video_stills/**/*.jpg`);

input.pipe(thumbsup(1024)).pipe(gulp.dest(`${media}/thumbs/video_stills/1024`));
input.pipe(thumbsup(400)).pipe(gulp.dest(`${media}/thumbs/video_stills/400`));
input.pipe(thumbsup(256)).pipe(gulp.dest(`${media}/thumbs/video_stills/128`));

return input;
}

const videos = video_thumbs;

const all = gulp.series(blog, modules, styles, videos);

const auto = () => {
return gulp.watch(['./**/*.scss', './db/**/*.md'], all);
}

module.exports = {
clean, blog, videos, structure, all, styles, modules,
default: auto
};

console.log("!!! REMEMBER THAT YOU HAD TO HAND CREATE THE MODULES/INDEX.JSON");

+ 0
- 1
media/blog/index.json View File

@@ -1 +0,0 @@
[{"title":"A First Test","slug":"a-first-test","url":"/blog/posts/a-first-test.md","summary":"This is just a little test.","has_image":true,"name":"/blog/posts/a-first-test.md"}]

+ 14
- 0
media/blog/posts/01-first-blog-post.md View File

@@ -0,0 +1,14 @@
{
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"has_image": true,
"summary": "A new kind of blog post for a new kind of blog.",
"title": "A First Real Blog Post"
}

------
# The First Real Blog Post

I've been working on this new system for publishing and managing my courses for about a month. It's actually been in my mind for about two years, but only recently all the pieces of JavaScript technology clicked together to make it easily possible. In this first blog post I'm going to go through the features and talk about why I'm working on this live.



+ 0
- 1
media/modules/0-html/index.json View File

@@ -1 +0,0 @@
[{"author":"Zed A. Shaw","date":"Mar 25, 2020","module":"/modules/0-html","video":{"src":"/images/sample.mp4","poster":"/images/sample.jpg","preload":"none"},"name":"/modules/0-html/intro.md"}]

+ 21
- 0
media/modules/drawing_basics/exercises/01-sight-sized.md View File

@@ -0,0 +1,21 @@
{
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"module": "/modules/drawin_basics",
"id": 1,
"title": "Site Sized Drawing",
"completed": false,
"image": "/images/hero-holder.svg",
"summary": "The first lesson where you learn about basic drawing terms and how to do sight sized drawing.",
"video": {
"src": "/videos/drawing_basics/Drawing_Basics_01_Sight_Sized.webm",
"poster": "/thumbs/video_stills/400/drawing_basics/01_Sight_Sized.jpg",
"preload": "none"
}
}

------

# Sight Sized Drawing

This is the first exercise where you have some stuff.

media/blog/posts/a-first-test.md → media/modules/html_basics/exercises/01-intro.md View File

@@ -1,13 +1,23 @@
{
"title":"A First Test",
"slug":"a-first-test",
"url":"/blog/posts/a-first-test.md",
"summary":"This is just a little test.",
"has_image":true
"author": "Zed A. Shaw",
"date": "Mar 25, 2020",
"module": "/modules/html_basics",
"id": 2,
"title": "Basic Terminology",
"completed": false,
"image": "/images/hero-holder.svg",
"summary": "This will first introduce some basic terminology for HTML.",
"video": {
"src": "/images/sample.mp4",
"poster": "/images/sample.jpg",
"preload": "none"
}
}

------
# Test Out This Week's Progress
# Introduction

This is the first blog post about learn javascript the hard way's new site. Here are some of the highlights of our blog activities so far.

We've added a new and improved page on one of our blog posts to give you a little taste of what we do. Since we're giving you a first taste, we may leave you with some teaser videos in the meantime.

@@ -15,6 +25,8 @@ You can see the new site on https://learnjsthehardway.com/

## The New Design

Rather than the current approach of measuring average page load time and customising each page to meet various requests, we're going to define a new metric we're going to call FRONTEND-TIME.

FRONTEND-TIME will measure how long it takes the browser to render and navigate your page, irrespective of the request it receives. It's a measure of how fast your site is responding.

For example, if we wanted to test the responsiveness of a popular page-load with the current system, we could test it like this:
@@ -24,7 +36,6 @@ For example, if we wanted to test the responsiveness of a popular page-load with
Testing out how to include code in the remarkable system:

```javascript

import Raw from '../../components/Raw.svelte';
export let posts;
export let segment;
@@ -35,8 +46,6 @@ if(posts == undefined) {
}
```

------

## TODO

I need to make the following things work out of remarkable:

+ 45
- 0
notes/video_encode.md View File

@@ -0,0 +1,45 @@
# Notes on Video Encoding

I seem to lose my notes every time I figure this out so I'm doing them one more time.

## Bit Rate

The br is the desired file size / seconds to get your bit rate, but it has to be kilobits:
1 byte is 0.008 kbits, or 1/125 = 0.008

bytes / 125 / seconds == kbit rate

In a 30 second sample I had, ffprobe said the kbit rate was 135193 and the file is 506976894 in size, so the calculation works:

506976894 / 125 / 30 == 135193

You need to then subtract the audio bitrate. So if you I want to get the file to 1MB in size (with a 128k audio rate), I have to do this:

1024 * 1024 / 125 / 30 - 128 == 151kbits

To encode this way, you need to do two passes:

ffmpeg -y -i test.mov -c:v libx264 -b:v 151k -pix_fmt yuv420p -pass 1 -an -f mp4 /dev/null
ffmpeg -i test.mov -c:v libx264 -b:v 151k -pass 2 -c:a aac -b:a 128k -pix_fmt yuv420p result.mp4


ffmpeg -y -i test.mov -c:v libx264 -b:v 151k -pix_fmt yuv420p -pass 1 -an -f mp4 /dev/null
ffmpeg -y -i test.mov -c:v libx264 -b:v 151k -pass 2 -an -pix_fmt yuv420p /dev/null
ffmpeg -i test.mov -c:v libx264 -b:v 151k -pass 3 -c:a aac -b:a 128k -pix_fmt yuv420p result_3pass.mp4


You can get a sample to work with by doing:

fmpeg -i Drawing_Basics_01_Sight_Sized.mov -t 120 -vcodec copy -acodec copy test.mov


If you encode as Grass Valley HQX then you can encode crazy fast and use ffmpeg to crunch it, then throw the GV file out.

## Resources

This is a list of links I used to figure a lot of this out:

http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide
https://trac.ffmpeg.org/wiki/Encode/VP9
https://trac.ffmpeg.org/wiki/Encode/H.264


+ 992
- 1
package-lock.json
File diff suppressed because it is too large
View File


+ 10
- 0
package.json View File

@@ -50,18 +50,27 @@
"@rollup/plugin-replace": "^2.3.1",
"@rollup/pluginutils": "^3.0.8",
"chromedriver": "^80.0.1",
"deep-cleaner": "^1.2.1",
"depcheck": "^0.9.2",
"event-stream": "^4.0.1",
"faker": "^4.1.0",
"fast-exif": "^1.0.1",
"fast-glob": "^3.2.2",
"geckodriver": "^1.19.1",
"gulp": "^4.0.2",
"gulp-autoprefixer": "^7.0.1",
"gulp-clean-css": "^4.3.0",
"gulp-csscomb": "^3.1.0",
"gulp-debug": "^4.0.0",
"gulp-exec": "^4.0.0",
"gulp-gitignore": "^0.1.0",
"gulp-notify": "^3.2.0",
"gulp-rename": "^2.0.0",
"gulp-sass": "^4.0.2",
"jest": "^25.1.0",
"jest-transform-stub": "^2.0.0",
"multipipe": "^4.0.0",
"natural-sort": "^1.0.0",
"node-notifier": "^6.0.0",
"npm-check": "^5.9.2",
"npm-run-all": "^4.1.5",
@@ -72,6 +81,7 @@
"rollup-plugin-svelte": "^5.0.1",
"rollup-plugin-terser": "^5.3.0",
"sapper": "^0.27.10",
"sharp": "^0.25.2",
"svelte": "^3.20.1",
"svelte-preprocess": "^3.5.0"
}

+ 1
- 1
src/node_modules/docs.js View File

@@ -56,7 +56,7 @@ async function load(preload, url, base_url, options) {
// TODO: is there an error to catch here from res?
return render(raw_md, base_url);
} else {
preload.error(res.status, data.message);
preload.error(res.status, 'Failed to load.');
}
}


+ 3
- 0
src/routes/blog/[slug].svelte View File

@@ -21,6 +21,9 @@
</script>

<style>
#content {
margin-top: 1rem;
}
</style>



+ 4
- 4
src/routes/blog/index.svelte View File

@@ -4,9 +4,9 @@

<script context="module">
export async function preload({ params, query }) {
let res = await this.fetch(`blog/index.json`);
let posts = await res.json();
return { posts };
let res = await this.fetch(`api/blog/index.json`);
let blog = await res.json();
return { posts: blog.posts };
}
</script>

@@ -58,7 +58,7 @@
{/if}
<div class="card-header">
<div class="card-title h5"> <a rel="prefetch" href="blog/{post.slug}">{post.title}</a></div>
<div class="card-subtitle text-gray">Subtitle goes here. </div>
<div class="card-subtitle text-gray">{ post.subtitle }</div>
</div>
<div class="card-body" data-testid="blog-summary">
{@html post.summary}

src/routes/modules/[slug]/[exercise]/index.svelte → src/routes/modules/[slug]/[exercise].svelte View File

@@ -6,12 +6,12 @@

<script context="module">
import docs from 'docs';
import Video from '../../../../components/Video.svelte';
import Video from '../../../components/Video.svelte';

export async function preload({ params, query }) {
let data = docs.load(this,
`/modules/0-html/intro.md`,
`/modules/${params.exercise}/${params.slug}`,
`modules/${params.slug}/exercises/${params.exercise}.md`,
`/modules/${params.slug}/${params.exercise}`,
{ credentials: 'same-origin'});
data.module = params.slug;
data.exercise = params.exercise;
@@ -21,7 +21,7 @@

<script>
import { onMount } from 'svelte';
import Icon from '../../../../components/Icon.svelte';
import Icon from '../../../components/Icon.svelte';
export let content;
export let toc;
export let metadata;
@@ -52,9 +52,9 @@
<div class="divider text-center"></div>

{#if metadata.video }
<div class="nav-item">
<a href="/modules/{module}/{exercise}/#video">Video</a>
</div>
<div class="nav-item">
<a href="/modules/{module}/{exercise}/#video">Video</a>
</div>
{/if}

<div class="nav-item">

+ 15
- 28
src/routes/modules/[slug]/index.svelte View File

@@ -1,31 +1,18 @@
<script context="module">
export async function preload({params, query }) {
let res = await this.fetch(`api/modules/${params.slug}/index.json`);
let module = await res.json();
console.log("module", module);
return { module };
}
</script>


<script>
import Icon from '../../../components/Icon.svelte';

let active = true;
let i = 1;
let module = {
title: `Sample ${i}`,
slug: `a-module-test-${i}`,
subtitle: `A module ${i}`,
tag: i % 6,
id: i,
image: "/images/hero-holder.svg",
summary: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent risus leo, dictum in vehicula sit amet, feugiat tempus tellus. Duis quis sodales risus. Etiam euismod ornare consequat.',
completed: i % 10,
total: i + 12,
reserved: i % 4 === 0,
}

let exercises = [...Array(10).keys()].map(i => {
return {
title: `Sample Title ${i}`,
slug: `${i}-a-module-test`,
image: '/images/hero-holder.svg',
summary: `Sample summar ${i}`,
completed: i < 4,
id: i,
}
});
export let module;
console.log("MODULE CLIENT", module);
</script>

<style lang="scss">
@@ -47,7 +34,7 @@
<li class="nav-item active">
<a href="/modules/"><Icon name="corner-up-left" /> { module.title }</a>
</li>
{#each exercises as exercise}
{#each module.exercises as exercise}
<li class="nav-item">
<a href="/modules/{module.slug}/{exercise.slug}">{exercise.title}</a>
</li>
@@ -72,7 +59,7 @@

<div class="filter-body">
<div class="container">
{#each exercises as exercise, i}
{#each module.exercises as exercise}
<div class="filter-item card tall" data-tag="tag-{exercise.completed ? 1 : 2}">
<div class="card-image">
<img alt="{exercise.title} image" src="{exercise.image}" class="img-responsive">
@@ -87,7 +74,7 @@
</div>

<div class="card-footer">
<a href="/modules/{module.id}-{module.slug}/{exercise.slug}" class="btn btn-secondary">Start Exercise</a>
<a href="/modules/{module.slug}/{exercise.slug}" class="btn btn-secondary">Start Exercise</a>
</div>
</div>
{/each}

+ 19
- 17
src/routes/modules/index.svelte View File

@@ -1,25 +1,27 @@
<script context="module">
export async function preload({ params, query }) {
let res = await this.fetch(`api/modules/index.json`);
let json = await res.json();
let modules = json.map((module, i) => {
// temporary hack until I get this data in
module.tag = i % 6;
module.id = i;
module.image = '/images/hero-holder.svg';
module.completed = i % 10;
module.total = i + 12;
module.reserved = i % 4 === 0;
return module;
});

return { modules };
}
</script>

<script>
import Icon from '../../components/Icon.svelte';

let active = true;

let modules = [...Array(10).keys()].map((i) => {
return {
title: `Sample ${i}`,
slug: `a-module-test-${i}`,
subtitle: `A module ${i}`,
tag: i % 6,
id: i,
image: "/images/hero-holder.svg",
summary: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent risus leo, dictum in vehicula sit amet, feugiat tempus tellus. Duis quis sodales risus. Etiam euismod ornare consequat.',
completed: i % 10,
total: i + 12,
reserved: i % 4 === 0,
}
});
export let modules;
</script>

<style lang="scss">
@@ -129,9 +131,9 @@
<div class="card-footer">
{#if module.reserved }
<button class="btn btn-secondary">Return</button>
<a href="/modules/{module.id}-{module.slug}/" class="btn btn-success badge" data-badge="{module.completed}/{module.total}">Continue</a>
<a href="/modules/{module.slug}/" class="btn btn-success badge" data-badge="{module.completed}/{module.total}">Continue</a>
{:else}
<a href="/modules/{module.id}-{module.slug}/" class="btn btn-secondary">Start Course</a>
<a href="/modules/{module.slug}/" class="btn btn-secondary">Start Course</a>
{/if}
</div>
</div>

+ 1
- 0
static/api/blog/index.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","url":"/blog/","title":"Learn JS The Hard Way","subtitle":"A blog about teaching JS on the internet.","posts":[{"author":"Zed A. Shaw","date":"Mar 25, 2020","has_image":true,"summary":"A new kind of blog post for a new kind of blog.","title":"A First Real Blog Post","url":"/blog/01-first-blog-post.md","slug":"01-first-blog-post","self":"/api/blog/01-first-blog-post.json"}]}

+ 1
- 0
static/api/blog/posts/01-first-blog-post.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","date":"Mar 25, 2020","has_image":true,"summary":"A new kind of blog post for a new kind of blog.","title":"A First Real Blog Post","url":"/blog/01-first-blog-post.md","slug":"01-first-blog-post"}

+ 1
- 0
static/api/modules/drawing_basics/exercises/01-sight-sized.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","date":"Mar 25, 2020","module":"/modules/drawin_basics","id":1,"title":"Site Sized Drawing","completed":false,"image":"/images/hero-holder.svg","summary":"The first lesson where you learn about basic drawing terms and how to do sight sized drawing.","video":{"src":"/videos/drawing_basics/Drawing_Basics_01_Sight_Sized.webm","poster":"/thumbs/video_stills/400/drawing_basics/01_Sight_Sized.jpg","preload":"none"},"url":"/modules/01-sight-sized.md","slug":"01-sight-sized"}

+ 1
- 0
static/api/modules/drawing_basics/index.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","date":"Mar 25, 2020","url":"/modules/drawing_basics","slug":"drawing_basics","title":"Drawing Basics","tag":"what","subtitle":"Learn just enough drawing to be deadly.","summary":"An introduction to drawing with pencil and digitally to get an understanding of it for the web.","exercises":[{"author":"Zed A. Shaw","date":"Mar 25, 2020","module":"/modules/drawin_basics","id":1,"title":"Site Sized Drawing","completed":false,"image":"/images/hero-holder.svg","summary":"The first lesson where you learn about basic drawing terms and how to do sight sized drawing.","video":{"src":"/videos/drawing_basics/Drawing_Basics_01_Sight_Sized.webm","poster":"/thumbs/video_stills/400/drawing_basics/01_Sight_Sized.jpg","preload":"none"},"url":"/modules/01-sight-sized.md","slug":"01-sight-sized","self":"/api/modules/drawing_basics/01-sight-sized.json"}]}

+ 1
- 0
static/api/modules/html_basics/exercises/01-intro.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","date":"Mar 25, 2020","module":"/modules/html_basics","id":2,"title":"Basic Terminology","completed":false,"image":"/images/hero-holder.svg","summary":"This will first introduce some basic terminology for HTML.","video":{"src":"/images/sample.mp4","poster":"/images/sample.jpg","preload":"none"},"url":"/modules/01-intro.md","slug":"01-intro"}

+ 1
- 0
static/api/modules/html_basics/index.json View File

@@ -0,0 +1 @@
{"author":"Zed A. Shaw","date":"Mar 25, 2020","url":"/modules/html_basics","slug":"html_basics","title":"HTML Basics","tag":"what","subtitle":"Start here if you know nothing.","summary":"An introduction to basic HTML for the web.","exercises":[{"author":"Zed A. Shaw","date":"Mar 25, 2020","module":"/modules/html_basics","id":2,"title":"Basic Terminology","completed":false,"image":"/images/hero-holder.svg","summary":"This will first introduce some basic terminology for HTML.","video":{"src":"/images/sample.mp4","poster":"/images/sample.jpg","preload":"none"},"url":"/modules/01-intro.md","slug":"01-intro","self":"/api/modules/html_basics/01-intro.json"}]}

+ 23
- 0
static/api/modules/index.json View File

@@ -0,0 +1,23 @@
[ {
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/drawing_basics",
"slug": "drawing_basics",
"title": "Drawing Basics",
"tag": "what",
"subtitle": "Learn just enough drawing to be deadly.",
"summary": "An introduction to drawing with pencil and digitally to get an understanding of it for the web.",
"exercises": []
},
{
"author":"Zed A. Shaw",
"date":"Mar 25, 2020",
"url":"/modules/html_basics",
"slug": "html_basics",
"title": "HTML Basics",
"tag": "what",
"subtitle": "Start here if you know nothing.",
"summary": "An introduction to basic HTML for the web.",
"exercises": []
}
]

+ 11
- 0
tools/convert.sh View File

@@ -0,0 +1,11 @@
BV=800k
BA=128k
TARGET=$1
CRF=20

rm ffmpeg*pass*

../ffmpeg -y -i $TARGET -tune animation -preset veryslow -c:v libx264 -b:v $BV -crf $CRF -pix_fmt yuv420p -pass 1 -an -f mp4 /dev/null
../ffmpeg -y -i $TARGET -tune animation -preset veryslow -c:v libx264 -b:v $BV -crf $CRF -pass 2 -an -pix_fmt yuv420p -f mp4 /dev/null
../ffmpeg -y -i $TARGET -tune animation -preset veryslow -c:v libx264 -b:v $BV -crf $CRF -pass 3 -c:a aac -b:a $BA -pix_fmt yuv420p ${TARGET%%.*}.mp4


+ 11
- 0
tools/convert_webm.sh View File

@@ -0,0 +1,11 @@
BV=800k
BA=128k
CRF=20
TARGET=$1
THREADS=-threads 4 -tile-columns 3 -frame-parallel 1

#rm ffmpeg*pass*

#./ffmpeg -y -i $TARGET ${THREADS} -speed 4 -c:v libvpx-vp9 -b:v $BV -crf $CRF -pix_fmt yuv420p -pass 1 -an -f mp4 /dev/null
./ffmpeg -y -i $TARGET ${THREADS} -auto-alt-ref 1 -lag-in-frames 25 -speed 1 -c:v libvpx-vp9 -b:v $BV -crf $CRF -pass 2 -c:a libopus -b:a $BA -pix_fmt yuv420p ${TARGET%%.*}.webm


+ 140
- 22
tools/gulp.js View File

@@ -1,33 +1,151 @@
const fs = require('fs').promises;
const glob = require('fast-glob');
const { src, dest, series, watch, parallel } = require('gulp');
const fs = require('fs');
const path = require('path');
const pipe = require('multipipe');
const debug = require('gulp-debug');
const es = require('event-stream');
const Vinyl = require('vinyl');
const assert = require('assert');
const { log } = require('../lib/logging');
const sharp = require('sharp');
const exif = require('fast-exif');
const cleaner = require('deep-cleaner');
const exec = require('child_process').execSync;
const naturalSort = require('natural-sort');

const split = (raw_md) => {
let [metadata, ...body] = raw_md.split('------');
metadata = JSON.parse(metadata);
body = body.join('------');
return [metadata, body];
/* hack configs that fix webtorrent not doing what I say */

const script = (src_files, commands) => {
let r = pipe(...commands);

r.on('error', err => {
console.error("[ERROR]", err.message);
throw err;
});

return src(src_files).pipe(r).pipe(debug());
}

exports.mdindex = async (base, path, target) => {
let md_files = await glob(path);
let index = [];
const markdownJSON = (base) => {
return es.map((data, cb) => {
if(data.contents.includes('---')) {
let md = JSON.parse(data.contents.toString().split('---')[0])
md.url = `${base}/${data.basename}`;
md.slug = `${data.stem}`;
data.contents = new Buffer.from(JSON.stringify(md));
cb(null, data);
} else {
console.error(`File ${data.path} does not have a JSON metadata header before ---.`);
data.contents = Buffer.from(`{"url": "${base}/${data.basename}"}`);
cb(null, data);
}
});
}

log.debug("Processing files", md_files);
const jsonIndex = (base="", index="", merge=null, merge_target="") => {
assert(base, 'You must give a base URL ending in /');
assert(index, 'You must give a place to write the index.');
assert(merge !== null, "You must give a merge object.");
assert(merge_target, "You need a valid key for the merge target.");
assert(merge_target in merge, `You need a placeholder for ${merge_target} in ${index}`);
let files = [];

for(let name of md_files) {
let data = await fs.readFile(name);
let [metadata, body] = split(data.toString());
const queue = (file) => {
let json = JSON.parse(file.contents.toString());
json.slug = file.stem;
json.self = `${base}${file.stem}.json`;
files.push(json);
}

// super dumb but, whatever it works
metadata.name = name.slice(base.length);
index.push(metadata);
const end = () => {
merge[merge_target] = files.sort((a, b) => a.slug < b.slug);
// output in the correct order based on ID
fs.writeFileSync(index, JSON.stringify(merge));
}

let jbuf = Buffer.from(JSON.stringify(index));
await fs.writeFile(target, jbuf);
return es.through(queue, end);
}

const thumbsup = (size) => {
return es.map(async (data, cb) => {
let thumb = await sharp(data.path)
.resize(size)
.toBuffer();
let result = data.clone({contents: false});
result.contents = thumb;
cb(null, result);
});
}

const cover_format = (p) => {
return {
small: `${p.dir}/thumbs/128/${p.base}`,
medium: `${p.dir}/thumbs/400/${p.base}`,
large: `${p.dir}/thumbs/1024/${p.base}`,
}
}

const covers = () => {
return es.map((data, cb) => {
let md = JSON.parse(data.contents.toString());
let is_photos = 'photos' in md;

if(is_photos) {
for(let photo of md.photos) {
let p = path.parse(photo.url);
photo.covers = cover_format(p);
}
} else {
let p = path.parse(md.url);
md.covers = cover_format(p);
}

data.contents = Buffer.from(JSON.stringify(md));
cb(null, data);
});
}

const REMOVE_EXIF = [ 'MakerNote', 'UserComment'];

const exifJSON = () => {
return es.map((data, cb) => {
// TODO: need some kind of base path thing I think
let md = JSON.parse(data.contents.toString());
let done = (md) => {
cleaner(md, REMOVE_EXIF);
data.contents = Buffer.from(JSON.stringify(md));
cb(null, data);
}
// this is convoluted because event-stream doesn't support async functions

if('photos' in md) {
let waiting = md.photos.map(x => {
return exif.read('.' + x.url).then(exifdata => x.exif = exifdata);
});

Promise.all(waiting).then(values => done(md)).catch(console.error);
} else if('url' in md) {
exif.read('.' + md.url).then(exifdata => {
md.exif = exifdata;
done(md);
}).catch(console.error);
} else {
console.error(data.path, 'NOT A PHOTO OR AN ART!');
}

});
}

const ffprobeJSON = () => {
return es.map((data, cb) => {
let md = JSON.parse(data.contents.toString());
let out = exec(`ffprobe -v quiet -print_format json -show_format -show_streams .${md.url}`);
md.probe = JSON.parse(out);
data.contents = Buffer.from(JSON.stringify(md));
cb(null, data);
});
}

exports.log = log;
exports.split = split;
module.exports = {
script, markdownJSON, jsonIndex, thumbsup, exifJSON, ffprobeJSON,
covers
};

+ 3
- 0
tools/sync.sh View File

@@ -0,0 +1,3 @@
rsync -rtzv --exclude ".*sw*" --exclude node_modules --exclude public/torrents --delete -h --progress --inplace * deploy@zedshaw.art:zedshawart/

echo "YOU HAVE TO RUN TORRENTS ON THE SERVER"

+ 42
- 0
tools/torrents.prod.sh View File

@@ -0,0 +1,42 @@
#!/usr/bin/env bash

export WWW=https://zedshaw.art
export TRACKER=https://zedshaw.art

function video_torrents {
for i in $*
do
mktorrent -a "$TRACKER/tracker/announce" -o public/torrents/$(basename "${i%.*}").torrent -p -w $WWW/media/videos/$(basename $i) -l 15 $i || true
done
}

function art_torrents {
for i in $*
do
mktorrent -a "$TRACKER/tracker/announce" -o public/torrents/$(basename "${i%.*}").torrent -p -w $WWW/media/art/$(basename $i) $i -l 15 || true
done
}

function video_stills {
for i in $*
do
result=media/video_stills/$(basename "${i%.*}").jpg
if [ ! -f "${result}" ]
then
ffmpeg -n -ss 3 -i $i -vf "select=gt(scene\,0.4)" -frames:v 1 -vsync vfr -vf fps=fps=1/600 ${result}

if [ ! -f "${result}" ]
then
echo "--------------- SHORT VIDEO?"
ffmpeg -n -ss 00:00:01 -i $i -frames:v 1 ${result}
fi
fi
done
}

mkdir -p media/video_stills
mkdir -p public/torrents

video_torrents media/videos/*
art_torrents media/art/*
video_stills media/videos/*

+ 48
- 0
tools/torrents.sh View File

@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -e

export WWW=http://localhost:5000
export TRACKER=ws://localhost:8000/tracker/announce

function torrents {
for i in $*
do
torrent=public/torrents/$(basename "${i%.*}").torrent

if [ ! -f "${torrent}" ]
then
mktorrent --verbose -a "$TRACKER" -o "${torrent}" -p -l 15 $i || true
fi
done
}

function video_stills {
for i in $*
do
dname=$( echo $i | cut -d / -f 3- | rev | cut -d / -f 2- | rev )
fname=$(basename $i .mp4)
result="media/video_stills${dname}/${fname%%.*}.jpg"

echo "Converting $i to $result in directory $dname"

mkdir -p "media/video_stills/${dname}"

if [ ! -f "${result}" ]
then
ffmpeg -n -i $i -ss 00:00:14 -frames:v 1 -vsync vfr ${result}

if [ ! -f "${result}" ]
then
echo "--------------- SHORT VIDEO?"
ffmpeg -ss 00:00:01 -i $i -frames:v 1 ${result}
fi
fi
done
}

mkdir -p media/video_stills
mkdir -p public/torrents

# torrents media/videos/drawing_basics/*
video_stills `find media/videos/ -name "*.mp4" -print`


Loading…
Cancel
Save