summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
Diffstat
-rw-r--r--README.md74
-rw-r--r--docker-compose.yml12
-rw-r--r--docker/Dockerfile4
-rw-r--r--docker/Dockerfile.postgres12
-rwxr-xr-xdocker/entrypoint.postgres.sh30
-rwxr-xr-xdocker/init-invidious-db.sh16
-rw-r--r--shard.yml6
-rw-r--r--spec/helpers_spec.cr2
-rw-r--r--src/invidious.cr209
-rw-r--r--src/invidious/channels.cr204
-rw-r--r--src/invidious/comments.cr69
-rw-r--r--src/invidious/helpers/helpers.cr416
-rw-r--r--src/invidious/helpers/jobs.cr2
-rw-r--r--src/invidious/helpers/macros.cr80
-rw-r--r--src/invidious/helpers/patch_mapping.cr166
-rw-r--r--src/invidious/mixes.cr60
-rw-r--r--src/invidious/playlists.cr333
-rw-r--r--src/invidious/search.cr80
-rw-r--r--src/invidious/users.cr333
-rw-r--r--src/invidious/videos.cr141
20 files changed, 1070 insertions, 1179 deletions
diff --git a/README.md b/README.md
index 478f2c65..6719e814 100644
--- a/README.md
+++ b/README.md
@@ -4,11 +4,9 @@
## Invidious is an alternative front-end to YouTube
+- [Copylefted libre software](https://github.com/iv-org/invidious) (AGPLv3+ licensed)
- Audio-only mode (and no need to keep window open on mobile)
-- [Free software](https://github.com/omarroth/invidious) (AGPLv3 licensed)
-- No ads
-- No need to create a Google account to save subscriptions
-- Lightweight (homepage is ~4 KB compressed)
+- Lightweight (the homepage is ~4 KB compressed)
- Tools for managing subscriptions:
- Only show unseen videos
- Only show latest (or latest unseen) video from each channel
@@ -18,11 +16,15 @@
- Dark mode
- Embed support
- Set default player options (speed, quality, autoplay, loop)
-- Does not require JS to play videos
-- Support for Reddit comments in place of YT comments
+- Support for Reddit comments in place of YouTube comments
- Import/Export subscriptions, watch history, preferences
+- [Developer API](https://github.com/iv-org/invidious/wiki/API)
- Does not use any of the official YouTube APIs
-- Developer [API](https://github.com/omarroth/invidious/wiki/API)
+- Does not require JavaScript to play videos
+- No need to create a Google account to save subscriptions
+- No ads
+- No CoC
+- No CLA
Liberapay: https://liberapay.com/omarroth
BTC: 356DpZyMXu6rYd55Yqzjs29n79kGKWcYrY
@@ -30,7 +32,7 @@ BCH: qq4ptclkzej5eza6a50et5ggc58hxsq5aylqut2npk
## Invidious Instances
-See [Invidious Instances](https://github.com/omarroth/invidious/wiki/Invidious-Instances) for a full list of publicly available instances.
+[Public instances](https://github.com/iv-org/invidious/wiki/Invidious-Instances) are to be found in this list.
### Official Instances
@@ -48,7 +50,7 @@ See [Invidious Instances](https://github.com/omarroth/invidious/wiki/Invidious-I
## Installation
-See [Invidious-Updater](https://github.com/tmiland/Invidious-Updater) for a self-contained script that can automatically install and update Invidious.
+[Invidious-Updater](https://github.com/tmiland/Invidious-Updater) is a self-contained script that can automatically install and update Invidious.
### Docker:
@@ -58,7 +60,7 @@ See [Invidious-Updater](https://github.com/tmiland/Invidious-Updater) for a self
$ docker-compose up
```
-And visit `localhost:3000` in your browser.
+Then visit `localhost:3000` in your browser.
#### Rebuild cluster:
@@ -75,7 +77,7 @@ $ docker-compose build
### Linux:
-#### Install dependencies
+#### Install the dependencies
```bash
# Arch Linux
@@ -91,16 +93,16 @@ $ sudo apt-get update
$ sudo apt install crystal libssl-dev libxml2-dev libyaml-dev libgmp-dev libreadline-dev postgresql librsvg2-bin libsqlite3-dev
```
-#### Add invidious user and clone repository
+#### Add an Invidious user and clone the repository
```bash
$ useradd -m invidious
$ sudo -i -u invidious
-$ git clone https://github.com/omarroth/invidious
+$ git clone https://github.com/iv-org/invidious
$ exit
```
-#### Setup PostgresSQL
+#### Set up PostgresSQL
```bash
$ sudo systemctl enable postgresql
@@ -120,7 +122,7 @@ $ psql invidious kemal < /home/invidious/invidious/config/sql/playlist_videos.sq
$ exit
```
-#### Setup Invidious
+#### Set up Invidious
```bash
$ sudo -i -u invidious
@@ -154,15 +156,15 @@ minsize 1048576
$ sudo chmod 0644 /etc/logrotate.d/invidious.logrotate
```
-### OSX:
+### macOS:
```bash
# Install dependencies
$ brew update
$ brew install shards crystal postgres imagemagick librsvg
-# Clone repository and setup postgres database
-$ git clone https://github.com/omarroth/invidious
+# Clone the repository and set up a PostgreSQL database
+$ git clone https://github.com/iv-org/invidious
$ cd invidious
$ brew services start postgresql
$ psql -c "CREATE ROLE kemal WITH PASSWORD 'kemal';" # Change 'kemal' here to a stronger password, and update `password` in config/config.yml
@@ -178,14 +180,14 @@ $ psql invidious kemal < config/sql/privacy.sql
$ psql invidious kemal < config/sql/playlists.sql
$ psql invidious kemal < config/sql/playlist_videos.sql
-# Setup Invidious
+# Set up Invidious
$ shards update && shards install
$ crystal build src/invidious.cr --release
```
## Update Invidious
-You can see how to update Invidious [here](https://github.com/omarroth/invidious/wiki/Updating).
+Instructions are available in the [updating guide](https://github.com/iv-org/invidious/wiki/Updating).
## Usage:
@@ -216,40 +218,34 @@ $ ./sentry
## Documentation
-[Documentation](https://github.com/omarroth/invidious/wiki) can be found in the wiki.
+[Documentation](https://github.com/iv-org/invidious/wiki) can be found in the wiki.
## Extensions
-[Extensions](https://github.com/omarroth/invidious/wiki/Extensions) can be found in the wiki, as well as documentation for integrating it into other projects.
+[Extensions](https://github.com/iv-org/invidious/wiki/Extensions) can be found in the wiki, as well as documentation for integrating it into other projects.
## Made with Invidious
-- [FreeTube](https://github.com/FreeTubeApp/FreeTube): An Open Source YouTube app for privacy.
-- [CloudTube](https://cadence.moe/cloudtube/subscriptions): A JS-rich alternate YouTube player
+- [FreeTube](https://github.com/FreeTubeApp/FreeTube): A libre software YouTube app for privacy.
+- [CloudTube](https://cadence.moe/cloudtube/subscriptions): A JavaScript-rich alternate YouTube player
- [PeerTubeify](https://gitlab.com/Cha_deL/peertubeify): On YouTube, displays a link to the same video on PeerTube, if it exists.
-- [MusicPiped](https://github.com/deep-gaurav/MusicPiped): A materialistic music player that streams music from YouTube.
-- [LapisTube](https://github.com/blubbll/lapis-tube): A fancy and advanced (experimental) YouTube frontend. Combined streams & custom YT features.
+- [MusicPiped](https://github.com/deep-gaurav/MusicPiped): A material design music player that streams music from YouTube.
+- [LapisTube](https://github.com/blubbll/lapis-tube): A fancy and advanced (experimental) YouTube front-end. Combined streams & custom YT features.
## Contributing
-1. Fork it ( https://github.com/omarroth/invidious/fork )
+1. Fork it ( https://github.com/iv-org/invidious/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
-5. Create a new Pull Request
+5. Create a new pull request
-## Contact
-
-Feel free to send an email to omarroth@protonmail.com or join our [Matrix Server](https://riot.im/app/#/room/#invidious:matrix.org), or #invidious on Freenode.
+#### Translation
-You can also view release notes on the [releases](https://github.com/omarroth/invidious/releases) page or in the CHANGELOG.md included in the repository.
+- Log in with an account you have elsewhere, or register an account and start translating at [Hosted Weblate](https://hosted.weblate.org/projects/invidious/).
-## License
+## Contact
-[![GNU AGPLv3 Image](https://www.gnu.org/graphics/agplv3-155x51.png)](http://www.gnu.org/licenses/agpl-3.0.en.html)
+Feel free to send an e-mail to omarroth@protonmail.com or join our [Matrix server](https://riot.im/app/#/room/#invidious:matrix.org), or #invidious on freenode.
-Invidious is Free Software: You can use, study share and improve it at your
-will. Specifically you can redistribute and/or modify it under the terms of the
-[GNU Affero General Public License](https://www.gnu.org/licenses/agpl.html) as
-published by the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
+You can also read the release notes on the [page of releases](https://github.com/iv-org/invidious/releases) [CHANGELOG.md](https://github.com/iv-org/invidious/blob/master/CHANGELOG.md) included in the repository.
diff --git a/docker-compose.yml b/docker-compose.yml
index 6a8612ac..bc292c53 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,14 +1,18 @@
version: '3'
services:
postgres:
- build:
- context: .
- dockerfile: docker/Dockerfile.postgres
+ image: postgres:10
restart: unless-stopped
volumes:
- postgresdata:/var/lib/postgresql/data
+ - ./config/sql:/config/sql
+ - ./docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh
+ environment:
+ POSTGRES_DB: invidious
+ POSTGRES_PASSWORD: kemal
+ POSTGRES_USER: kemal
healthcheck:
- test: ["CMD", "pg_isready", "-U", "postgres"]
+ test: ["CMD", "pg_isready", "-U", "postgres"]
invidious:
build:
context: .
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 31760d71..96f844fe 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -1,4 +1,4 @@
-FROM crystallang/crystal:0.35.0-alpine AS builder
+FROM crystallang/crystal:0.35.1-alpine AS builder
RUN apk add --no-cache curl sqlite-static
WORKDIR /invidious
COPY ./shard.yml ./shard.yml
@@ -8,7 +8,7 @@ RUN shards update && shards install && \
# https://github.com/omarroth/lsquic-alpine/blob/master/APKBUILD,
# https://github.com/omarroth/lsquic.cr/issues/1#issuecomment-631610081
# for details building static lib
- curl -Lo ./lib/lsquic/src/lsquic/ext/liblsquic.a https://omar.yt/lsquic/liblsquic.a
+ curl -Lo ./lib/lsquic/src/lsquic/ext/liblsquic.a https://omar.yt/lsquic/liblsquic-v2.18.1.a
COPY ./src/ ./src/
# TODO: .git folder is required for building – this is destructive.
# See definition of CURRENT_BRANCH, CURRENT_COMMIT and CURRENT_VERSION.
diff --git a/docker/Dockerfile.postgres b/docker/Dockerfile.postgres
deleted file mode 100644
index 3b25b802..00000000
--- a/docker/Dockerfile.postgres
+++ /dev/null
@@ -1,12 +0,0 @@
-FROM postgres:10
-
-ENV POSTGRES_USER postgres
-# Do not require a PostgreSQL superuser password.
-# See https://github.com/docker-library/postgres/issues/681.
-ENV POSTGRES_HOST_AUTH_METHOD trust
-
-ADD ./config/sql /config/sql
-ADD ./docker/entrypoint.postgres.sh /entrypoint.sh
-
-ENTRYPOINT [ "/entrypoint.sh" ]
-CMD [ "postgres" ]
diff --git a/docker/entrypoint.postgres.sh b/docker/entrypoint.postgres.sh
deleted file mode 100755
index be6f6782..00000000
--- a/docker/entrypoint.postgres.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-
-CMD="$@"
-if [ ! -f /var/lib/postgresql/data/setupFinished ]; then
- echo "### first run - setting up invidious database"
- /usr/local/bin/docker-entrypoint.sh postgres &
- sleep 10
- until runuser -l postgres -c 'pg_isready' 2>/dev/null; do
- >&2 echo "### Postgres is unavailable - waiting"
- sleep 5
- done
- >&2 echo "### importing table schemas"
- su postgres -c 'createdb invidious'
- su postgres -c 'psql -c "CREATE USER kemal WITH PASSWORD '"'kemal'"'"'
- su postgres -c 'psql invidious kemal < config/sql/channels.sql'
- su postgres -c 'psql invidious kemal < config/sql/videos.sql'
- su postgres -c 'psql invidious kemal < config/sql/channel_videos.sql'
- su postgres -c 'psql invidious kemal < config/sql/users.sql'
- su postgres -c 'psql invidious kemal < config/sql/session_ids.sql'
- su postgres -c 'psql invidious kemal < config/sql/nonces.sql'
- su postgres -c 'psql invidious kemal < config/sql/annotations.sql'
- su postgres -c 'psql invidious kemal < config/sql/playlists.sql'
- su postgres -c 'psql invidious kemal < config/sql/playlist_videos.sql'
- touch /var/lib/postgresql/data/setupFinished
- echo "### invidious database setup finished"
- exit
-fi
-
-echo "running postgres /usr/local/bin/docker-entrypoint.sh $CMD"
-exec /usr/local/bin/docker-entrypoint.sh $CMD
diff --git a/docker/init-invidious-db.sh b/docker/init-invidious-db.sh
new file mode 100755
index 00000000..6c8ad3a7
--- /dev/null
+++ b/docker/init-invidious-db.sh
@@ -0,0 +1,16 @@
+#!/bin/sh
+set -eou pipefail
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
+ CREATE USER postgres;
+EOSQL
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/channels.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/videos.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/channel_videos.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/users.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/session_ids.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/nonces.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/annotations.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/playlists.sql
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < config/sql/playlist_videos.sql
diff --git a/shard.yml b/shard.yml
index 2c1e54aa..952020fb 100644
--- a/shard.yml
+++ b/shard.yml
@@ -17,7 +17,7 @@ dependencies:
version: ~> 0.16.0
kemal:
github: kemalcr/kemal
- branch: master
+ commit: dfe7dca08f4c9a9456d6132af5f6b59fcd6865e4
pool:
github: ysbaddaden/pool
version: ~> 0.2.3
@@ -26,8 +26,8 @@ dependencies:
version: ~> 0.1.2
lsquic:
github: omarroth/lsquic.cr
- branch: dev
+ version: ~> 2.18.1
-crystal: 0.35.0
+crystal: 0.35.1
license: AGPLv3
diff --git a/spec/helpers_spec.cr b/spec/helpers_spec.cr
index fe16e716..a8a3c6ce 100644
--- a/spec/helpers_spec.cr
+++ b/spec/helpers_spec.cr
@@ -12,6 +12,8 @@ require "../src/invidious/search"
require "../src/invidious/trending"
require "../src/invidious/users"
+CONFIG = Config.from_yaml(File.open("config/config.yml"))
+
describe "Helper" do
describe "#produce_channel_videos_url" do
it "correctly produces url for requesting page `x` of a channel's videos" do
diff --git a/src/invidious.cr b/src/invidious.cr
index c95c6419..10722162 100644
--- a/src/invidious.cr
+++ b/src/invidious.cr
@@ -1203,17 +1203,17 @@ post "/playlist_ajax" do |env|
end
end
- playlist_video = PlaylistVideo.new(
- title: video.title,
- id: video.id,
- author: video.author,
- ucid: video.ucid,
+ playlist_video = PlaylistVideo.new({
+ title: video.title,
+ id: video.id,
+ author: video.author,
+ ucid: video.ucid,
length_seconds: video.length_seconds,
- published: video.published,
- plid: playlist_id,
- live_now: video.live_now,
- index: Random::Secure.rand(0_i64..Int64::MAX)
- )
+ published: video.published,
+ plid: playlist_id,
+ live_now: video.live_now,
+ index: Random::Secure.rand(0_i64..Int64::MAX),
+ })
video_array = playlist_video.to_a
args = arg_array(video_array)
@@ -1839,8 +1839,8 @@ post "/login" do |env|
sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
user, sid = create_user(sid, email, password)
user_array = user.to_a
+ user_array[4] = user_array[4].to_json # User preferences
- user_array[4] = user_array[4].to_json
args = arg_array(user_array)
PG_DB.exec("INSERT INTO users VALUES (#{args})", args: user_array)
@@ -2427,18 +2427,39 @@ get "/subscription_manager" do |env|
end
subscriptions = PG_DB.query_all("SELECT * FROM channels WHERE id = ANY(#{values})", as: InvidiousChannel)
-
subscriptions.sort_by! { |channel| channel.author.downcase }
if action_takeout
if format == "json"
env.response.content_type = "application/json"
env.response.headers["content-disposition"] = "attachment"
- next {
- "subscriptions" => user.subscriptions,
- "watch_history" => user.watched,
- "preferences" => user.preferences,
- }.to_json
+ playlists = PG_DB.query_all("SELECT * FROM playlists WHERE author = $1 AND id LIKE 'IV%' ORDER BY created", user.email, as: InvidiousPlaylist)
+
+ next JSON.build do |json|
+ json.object do
+ json.field "subscriptions", user.subscriptions
+ json.field "watch_history", user.watched
+ json.field "preferences", user.preferences
+ json.field "playlists" do
+ json.array do
+ playlists.each do |playlist|
+ json.object do
+ json.field "title", playlist.title
+ json.field "description", html_to_content(playlist.description_html)
+ json.field "privacy", playlist.privacy.to_s
+ json.field "videos" do
+ json.array do
+ PG_DB.query_all("SELECT id FROM playlist_videos WHERE plid = $1 ORDER BY array_position($2, index) LIMIT 500", playlist.id, playlist.index, as: String).each do |video_id|
+ json.string video_id
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+ end
else
env.response.content_type = "application/xml"
env.response.headers["content-disposition"] = "attachment"
@@ -2498,41 +2519,11 @@ post "/data_control" do |env|
if user
user = user.as(User)
- spawn do
- # Since import can take a while, if we're not done after 20 seconds
- # push out content to prevent timeout.
-
- # Interesting to note is that Chrome will try to render before the content has finished loading,
- # which is why we include a loading icon. Firefox and its derivatives will not see this page,
- # instead redirecting immediately once the connection has closed.
-
- # https://stackoverflow.com/q/2091239 is helpful but not directly applicable here.
-
- sleep 20.seconds
- env.response.puts %(<meta http-equiv="refresh" content="0; url=#{referer}">)
- env.response.puts %(<link rel="stylesheet" href="/css/ionicons.min.css?v=#{ASSET_COMMIT}">)
- env.response.puts %(<link rel="stylesheet" href="/css/default.css?v=#{ASSET_COMMIT}">)
- if env.get("preferences").as(Preferences).dark_mode == "dark"
- env.response.puts %(<link rel="stylesheet" href="/css/darktheme.css?v=#{ASSET_COMMIT}">)
- else
- env.response.puts %(<link rel="stylesheet" href="/css/lighttheme.css?v=#{ASSET_COMMIT}">)
- end
- env.response.puts %(<h3><div class="loading"><i class="icon ion-ios-refresh"></i></div></h3>)
- env.response.flush
-
- loop do
- env.response.puts %(<!-- keepalive #{Time.utc.to_unix} -->)
- env.response.flush
-
- sleep (20 + rand(11)).seconds
- end
- end
+ # TODO: Find a way to prevent browser timeout
HTTP::FormData.parse(env.request) do |part|
body = part.body.gets_to_end
- if body.empty?
- next
- end
+ next if body.empty?
# TODO: Unify into single import based on content-type
case part.name
@@ -2555,9 +2546,55 @@ post "/data_control" do |env|
end
if body["preferences"]?
- user.preferences = Preferences.from_json(body["preferences"].to_json, user.preferences)
+ user.preferences = Preferences.from_json(body["preferences"].to_json)
PG_DB.exec("UPDATE users SET preferences = $1 WHERE email = $2", user.preferences.to_json, user.email)
end
+
+ if playlists = body["playlists"]?.try &.as_a?
+ playlists.each do |item|
+ title = item["title"]?.try &.as_s?.try &.delete("<>")
+ description = item["description"]?.try &.as_s?.try &.delete("\r")
+ privacy = item["privacy"]?.try &.as_s?.try { |privacy| PlaylistPrivacy.parse? privacy }
+
+ next if !title
+ next if !description
+ next if !privacy
+
+ playlist = create_playlist(PG_DB, title, privacy, user)
+ PG_DB.exec("UPDATE playlists SET description = $1 WHERE id = $2", description, playlist.id)
+
+ videos = item["videos"]?.try &.as_a?.try &.each_with_index do |video_id, idx|
+ raise "Playlist cannot have more than 500 videos" if idx > 500
+
+ video_id = video_id.try &.as_s?
+ next if !video_id
+
+ begin
+ video = get_video(video_id, PG_DB)
+ rescue ex
+ next
+ end
+
+ playlist_video = PlaylistVideo.new({
+ title: video.title,
+ id: video.id,
+ author: video.author,
+ ucid: video.ucid,
+ length_seconds: video.length_seconds,
+ published: video.published,
+ plid: playlist.id,
+ live_now: video.live_now,
+ index: Random::Secure.rand(0_i64..Int64::MAX),
+ })
+
+ video_array = playlist_video.to_a
+ args = arg_array(video_array)
+
+ PG_DB.exec("INSERT INTO playlist_videos VALUES (#{args})", args: video_array)
+ PG_DB.exec("UPDATE playlists SET index = array_append(index, $1), video_count = cardinality(index), updated = $2 WHERE id = $3", playlist_video.index, Time.utc, playlist.id)
+ end
+ end
+ end
when "import_youtube"
subscriptions = XML.parse(body)
user.subscriptions += subscriptions.xpath_nodes(%q(//outline[@type="rss"])).map do |channel|
@@ -3119,20 +3156,20 @@ get "/feed/channel/:ucid" do |env|
description_html = entry.xpath_node("group/description").not_nil!.to_s
views = entry.xpath_node("group/community/statistics").not_nil!.["views"].to_i64
- SearchVideo.new(
- title: title,
- id: video_id,
- author: author,
- ucid: ucid,
- published: published,
- views: views,
- description_html: description_html,
- length_seconds: 0,
- live_now: false,
- paid: false,
- premium: false,
- premiere_timestamp: nil
- )
+ SearchVideo.new({
+ title: title,
+ id: video_id,
+ author: author,
+ ucid: ucid,
+ published: published,
+ views: views,
+ description_html: description_html,
+ length_seconds: 0,
+ live_now: false,
+ paid: false,
+ premium: false,
+ premiere_timestamp: nil,
+ })
end
XML.build(indent: " ", encoding: "UTF-8") do |xml|
@@ -3362,18 +3399,18 @@ post "/feed/webhook/:token" do |env|
}.to_json
PG_DB.exec("NOTIFY notifications, E'#{payload}'")
- video = ChannelVideo.new(
- id: id,
- title: video.title,
- published: published,
- updated: updated,
- ucid: video.ucid,
- author: author,
- length_seconds: video.length_seconds,
- live_now: video.live_now,
+ video = ChannelVideo.new({
+ id: id,
+ title: video.title,
+ published: published,
+ updated: updated,
+ ucid: video.ucid,
+ author: author,
+ length_seconds: video.length_seconds,
+ live_now: video.live_now,
premiere_timestamp: video.premiere_timestamp,
- views: video.views,
- )
+ views: video.views,
+ })
PG_DB.query_all("UPDATE users SET feed_needs_update = true, notifications = array_append(notifications, $1) \
WHERE updated < $2 AND $3 = ANY(subscriptions) AND $1 <> ALL(notifications)",
@@ -4631,7 +4668,7 @@ post "/api/v1/auth/preferences" do |env|
user = env.get("user").as(User)
begin
- preferences = Preferences.from_json(env.request.body || "{}", user.preferences)
+ preferences = Preferences.from_json(env.request.body || "{}")
rescue
preferences = user.preferences
end
@@ -4885,17 +4922,17 @@ post "/api/v1/auth/playlists/:plid/videos" do |env|
next error_message
end
- playlist_video = PlaylistVideo.new(
- title: video.title,
- id: video.id,
- author: video.author,
- ucid: video.ucid,
+ playlist_video = PlaylistVideo.new({
+ title: video.title,
+ id: video.id,
+ author: video.author,
+ ucid: video.ucid,
length_seconds: video.length_seconds,
- published: video.published,
- plid: plid,
- live_now: video.live_now,
- index: Random::Secure.rand(0_i64..Int64::MAX)
- )
+ published: video.published,
+ plid: plid,
+ live_now: video.live_now,
+ index: Random::Secure.rand(0_i64..Int64::MAX),
+ })
video_array = playlist_video.to_a
args = arg_array(video_array)
diff --git a/src/invidious/channels.cr b/src/invidious/channels.cr
index e7bcf00e..da062755 100644
--- a/src/invidious/channels.cr
+++ b/src/invidious/channels.cr
@@ -1,14 +1,27 @@
struct InvidiousChannel
- db_mapping({
- id: String,
- author: String,
- updated: Time,
- deleted: Bool,
- subscribed: Time?,
- })
+ include DB::Serializable
+
+ property id : String
+ property author : String
+ property updated : Time
+ property deleted : Bool
+ property subscribed : Time?
end
struct ChannelVideo
+ include DB::Serializable
+
+ property id : String
+ property title : String
+ property published : Time
+ property updated : Time
+ property ucid : String
+ property author : String
+ property length_seconds : Int32 = 0
+ property live_now : Bool = false
+ property premiere_timestamp : Time? = nil
+ property views : Int64? = nil
+
def to_json(locale, json : JSON::Builder)
json.object do
json.field "type", "shortVideo"
@@ -84,49 +97,36 @@ struct ChannelVideo
end
end
end
-
- db_mapping({
- id: String,
- title: String,
- published: Time,
- updated: Time,
- ucid: String,
- author: String,
- length_seconds: {type: Int32, default: 0},
- live_now: {type: Bool, default: false},
- premiere_timestamp: {type: Time?, default: nil},
- views: {type: Int64?, default: nil},
- })
end
struct AboutRelatedChannel
- db_mapping({
- ucid: String,
- author: String,
- author_url: String,
- author_thumbnail: String,
- })
+ include DB::Serializable
+
+ property ucid : String
+ property author : String
+ property author_url : String
+ property author_thumbnail : String
end
# TODO: Refactor into either SearchChannel or InvidiousChannel
struct AboutChannel
- db_mapping({
- ucid: String,
- author: String,
- auto_generated: Bool,
- author_url: String,
- author_thumbnail: String,
- banner: String?,
- description_html: String,
- paid: Bool,
- total_views: Int64,
- sub_count: Int32,
- joined: Time,
- is_family_friendly: Bool,
- allowed_regions: Array(String),
- related_channels: Array(AboutRelatedChannel),
- tabs: Array(String),
- })
+ include DB::Serializable
+
+ property ucid : String
+ property author : String
+ property auto_generated : Bool
+ property author_url : String
+ property author_thumbnail : String
+ property banner : String?
+ property description_html : String
+ property paid : Bool
+ property total_views : Int64
+ property sub_count : Int32
+ property joined : Time
+ property is_family_friendly : Bool
+ property allowed_regions : Array(String)
+ property related_channels : Array(AboutRelatedChannel)
+ property tabs : Array(String)
end
class ChannelRedirect < Exception
@@ -248,18 +248,18 @@ def fetch_channel(ucid, db, pull_all_videos = true, locale = nil)
premiere_timestamp = channel_video.try &.premiere_timestamp
- video = ChannelVideo.new(
- id: video_id,
- title: title,
- published: published,
- updated: Time.utc,
- ucid: ucid,
- author: author,
- length_seconds: length_seconds,
- live_now: live_now,
+ video = ChannelVideo.new({
+ id: video_id,
+ title: title,
+ published: published,
+ updated: Time.utc,
+ ucid: ucid,
+ author: author,
+ length_seconds: length_seconds,
+ live_now: live_now,
premiere_timestamp: premiere_timestamp,
- views: views,
- )
+ views: views,
+ })
emails = db.query_all("UPDATE users SET notifications = array_append(notifications, $1) \
WHERE updated < $2 AND $3 = ANY(subscriptions) AND $1 <> ALL(notifications) RETURNING email",
@@ -298,18 +298,18 @@ def fetch_channel(ucid, db, pull_all_videos = true, locale = nil)
videos = extract_videos(initial_data.as_h, author, ucid)
count = videos.size
- videos = videos.map { |video| ChannelVideo.new(
- id: video.id,
- title: video.title,
- published: video.published,
- updated: Time.utc,
- ucid: video.ucid,
- author: video.author,
- length_seconds: video.length_seconds,
- live_now: video.live_now,
+ videos = videos.map { |video| ChannelVideo.new({
+ id: video.id,
+ title: video.title,
+ published: video.published,
+ updated: Time.utc,
+ ucid: video.ucid,
+ author: video.author,
+ length_seconds: video.length_seconds,
+ live_now: video.live_now,
premiere_timestamp: video.premiere_timestamp,
- views: video.views
- ) }
+ views: video.views,
+ }) }
videos.each do |video|
ids << video.id
@@ -352,7 +352,13 @@ def fetch_channel(ucid, db, pull_all_videos = true, locale = nil)
db.exec("DELETE FROM channel_videos * WHERE NOT id = ANY ('{#{ids.map { |id| %("#{id}") }.join(",")}}') AND ucid = $1", ucid)
end
- channel = InvidiousChannel.new(ucid, author, Time.utc, false, nil)
+ channel = InvidiousChannel.new({
+ id: ucid,
+ author: author,
+ updated: Time.utc,
+ deleted: false,
+ subscribed: nil,
+ })
return channel
end
@@ -395,12 +401,12 @@ def produce_channel_videos_url(ucid, page = 1, auto_generated = nil, sort_by = "
"80226972:embedded" => {
"2:string" => ucid,
"3:base64" => {
- "2:string" => "videos",
- "6:varint": 2_i64,
- "7:varint": 1_i64,
- "12:varint": 1_i64,
- "13:string": "",
- "23:varint": 0_i64,
+ "2:string" => "videos",
+ "6:varint" => 2_i64,
+ "7:varint" => 1_i64,
+ "12:varint" => 1_i64,
+ "13:string" => "",
+ "23:varint" => 0_i64,
},
},
}
@@ -444,12 +450,12 @@ def produce_channel_playlists_url(ucid, cursor, sort = "newest", auto_generated
"80226972:embedded" => {
"2:string" => ucid,
"3:base64" => {
- "2:string" => "playlists",
- "6:varint": 2_i64,
- "7:varint": 1_i64,
- "12:varint": 1_i64,
- "13:string": "",
- "23:varint": 0_i64,
+ "2:string" => "playlists",
+ "6:varint" => 2_i64,
+ "7:varint" => 1_i64,
+ "12:varint" => 1_i64,
+ "13:string" => "",
+ "23:varint" => 0_i64,
},
},
}
@@ -849,12 +855,12 @@ def get_about_info(ucid, locale)
related_author_thumbnail = node.xpath_node(%q(.//img)).try &.["data-thumb"]
related_author_thumbnail ||= ""
- AboutRelatedChannel.new(
- ucid: related_id,
- author: related_title,
- author_url: related_author_url,
+ AboutRelatedChannel.new({
+ ucid: related_id,
+ author: related_title,
+ author_url: related_author_url,
author_thumbnail: related_author_thumbnail,
- )
+ })
end
joined = about.xpath_node(%q(//span[contains(., "Joined")]))
@@ -876,23 +882,23 @@ def get_about_info(ucid, locale)
tabs = about.xpath_nodes(%q(//ul[@id="channel-navigation-menu"]/li/a/span)).map { |node| node.content.downcase }
- AboutChannel.new(
- ucid: ucid,
- author: author,
- auto_generated: auto_generated,
- author_url: author_url,
- author_thumbnail: author_thumbnail,
- banner: banner,
- description_html: description_html,
- paid: paid,
- total_views: total_views,
- sub_count: sub_count,
- joined: joined,
+ AboutChannel.new({
+ ucid: ucid,
+ author: author,
+ auto_generated: auto_generated,
+ author_url: author_url,
+ author_thumbnail: author_thumbnail,
+ banner: banner,
+ description_html: description_html,
+ paid: paid,
+ total_views: total_views,
+ sub_count: sub_count,
+ joined: joined,
is_family_friendly: is_family_friendly,
- allowed_regions: allowed_regions,
- related_channels: related_channels,
- tabs: tabs
- )
+ allowed_regions: allowed_regions,
+ related_channels: related_channels,
+ tabs: tabs,
+ })
end
def get_60_videos(ucid, author, page, auto_generated, sort_by = "newest")
diff --git a/src/invidious/comments.cr b/src/invidious/comments.cr
index 5490d2ea..407cef78 100644
--- a/src/invidious/comments.cr
+++ b/src/invidious/comments.cr
@@ -1,11 +1,23 @@
class RedditThing
- JSON.mapping({
- kind: String,
- data: RedditComment | RedditLink | RedditMore | RedditListing,
- })
+ include JSON::Serializable
+
+ property kind : String
+ property data : RedditComment | RedditLink | RedditMore | RedditListing
end
class RedditComment
+ include JSON::Serializable
+
+ property author : String
+ property body_html : String
+ property replies : RedditThing | String
+ property score : Int32
+ property depth : Int32
+ property permalink : String
+
+ @[JSON::Field(converter: RedditComment::TimeConverter)]
+ property created_utc : Time
+
module TimeConverter
def self.from_json(value : JSON::PullParser) : Time
Time.unix(value.read_float.to_i)
@@ -15,46 +27,33 @@ class RedditComment
json.number(value.to_unix)
end
end
-
- JSON.mapping({
- author: String,
- body_html: String,
- replies: RedditThing | String,
- score: Int32,
- depth: Int32,
- permalink: String,
- created_utc: {
- type: Time,
- converter: RedditComment::TimeConverter,
- },
- })
end
struct RedditLink
- JSON.mapping({
- author: String,
- score: Int32,
- subreddit: String,
- num_comments: Int32,
- id: String,
- permalink: String,
- title: String,
- })
+ include JSON::Serializable
+
+ property author : String
+ property score : Int32
+ property subreddit : String
+ property num_comments : Int32
+ property id : String
+ property permalink : String
+ property title : String
end
struct RedditMore
- JSON.mapping({
- children: Array(String),
- count: Int32,
- depth: Int32,
- })
+ include JSON::Serializable
+
+ property children : Array(String)
+ property count : Int32
+ property depth : Int32
end
class RedditListing
- JSON.mapping({
- children: Array(RedditThing),
- modhash: String,
- })
+ include JSON::Serializable
+
+ property children : Array(RedditThing)
+ property modhash : String
end
def fetch_youtube_comments(id, db, cursor, format, locale, thin_mode, region, sort_by = "top")
diff --git a/src/invidious/helpers/helpers.cr b/src/invidious/helpers/helpers.cr
index aaec19c5..56f856c0 100644
--- a/src/invidious/helpers/helpers.cr
+++ b/src/invidious/helpers/helpers.cr
@@ -1,219 +1,100 @@
require "./macros"
struct Nonce
- db_mapping({
- nonce: String,
- expire: Time,
- })
+ include DB::Serializable
+
+ property nonce : String
+ property expire : Time
end
struct SessionId
- db_mapping({
- id: String,
- email: String,
- issued: String,
- })
+ include DB::Serializable
+
+ property id : String
+ property email : String
+ property issued : String
end
struct Annotation
- db_mapping({
- id: String,
- annotations: String,
- })
+ include DB::Serializable
+
+ property id : String
+ property annotations : String
end
struct ConfigPreferences
- module StringToArray
- def self.to_json(value : Array(String), json : JSON::Builder)
- json.array do
- value.each do |element|
- json.string element
- end
- end
- end
-
- def self.from_json(value : JSON::PullParser) : Array(String)
- begin
- result = [] of String
- value.read_array do
- result << HTML.escape(value.read_string[0, 100])
- end
- rescue ex
- result = [HTML.escape(value.read_string[0, 100]), ""]
- end
-
- result
- end
-
- def self.to_yaml(value : Array(String), yaml : YAML::Nodes::Builder)
- yaml.sequence do
- value.each do |element|
- yaml.scalar element
- end
- end
- end
-
- def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : Array(String)
- begin
- unless node.is_a?(YAML::Nodes::Sequence)
- node.raise "Expected sequence, not #{node.class}"
- end
-
- result = [] of String
- node.nodes.each do |item|
- unless item.is_a?(YAML::Nodes::Scalar)
- node.raise "Expected scalar, not #{item.class}"
- end
-
- result << HTML.escape(item.value[0, 100])
- end
- rescue ex
- if node.is_a?(YAML::Nodes::Scalar)
- result = [HTML.escape(node.value[0, 100]), ""]
- else
- result = ["", ""]
- end
- end
-
- result
- end
- end
-
- module BoolToString
- def self.to_json(value : String, json : JSON::Builder)
- json.string value
- end
-
- def self.from_json(value : JSON::PullParser) : String
- begin
- result = value.read_string
-
- if result.empty?
- CONFIG.default_user_preferences.dark_mode
- else
- result
- end
- rescue ex
- if value.read_bool
- "dark"
- else
- "light"
- end
- end
- end
-
- def self.to_yaml(value : String, yaml : YAML::Nodes::Builder)
- yaml.scalar value
- end
-
- def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : String
- unless node.is_a?(YAML::Nodes::Scalar)
- node.raise "Expected scalar, not #{node.class}"
- end
-
- case node.value
- when "true"
- "dark"
- when "false"
- "light"
- when ""
- CONFIG.default_user_preferences.dark_mode
- else
- node.value
- end
- end
+ include YAML::Serializable
+
+ property annotations : Bool = false
+ property annotations_subscribed : Bool = false
+ property autoplay : Bool = false
+ property captions : Array(String) = ["", "", ""]
+ property comments : Array(String) = ["youtube", ""]
+ property continue : Bool = false
+ property continue_autoplay : Bool = true
+ property dark_mode : String = ""
+ property latest_only : Bool = false
+ property listen : Bool = false
+ property local : Bool = false
+ property locale : String = "en-US"
+ property max_results : Int32 = 40
+ property notifications_only : Bool = false
+ property player_style : String = "invidious"
+ property quality : String = "hd720"
+ property default_home : String = "Popular"
+ property feed_menu : Array(String) = ["Popular", "Trending", "Subscriptions", "Playlists"]
+ property related_videos : Bool = true
+ property sort : String = "published"
+ property speed : Float32 = 1.0_f32
+ property thin_mode : Bool = false
+ property unseen_only : Bool = false
+ property video_loop : Bool = false
+ property volume : Int32 = 100
+
+ def to_tuple
+ {% begin %}
+ {
+ {{*@type.instance_vars.map { |var| "#{var.name}: #{var.name}".id }}}
+ }
+ {% end %}
end
-
- yaml_mapping({
- annotations: {type: Bool, default: false},
- annotations_subscribed: {type: Bool, default: false},
- autoplay: {type: Bool, default: false},
- captions: {type: Array(String), default: ["", "", ""], converter: StringToArray},
- comments: {type: Array(String), default: ["youtube", ""], converter: StringToArray},
- continue: {type: Bool, default: false},
- continue_autoplay: {type: Bool, default: true},
- dark_mode: {type: String, default: "", converter: BoolToString},
- latest_only: {type: Bool, default: false},
- listen: {type: Bool, default: false},
- local: {type: Bool, default: false},
- locale: {type: String, default: "en-US"},
- max_results: {type: Int32, default: 40},
- notifications_only: {type: Bool, default: false},
- player_style: {type: String, default: "invidious"},
- quality: {type: String, default: "hd720"},
- default_home: {type: String, default: "Popular"},
- feed_menu: {type: Array(String), default: ["Popular", "Trending", "Subscriptions", "Playlists"]},
- related_videos: {type: Bool, default: true},
- sort: {type: String, default: "published"},
- speed: {type: Float32, default: 1.0_f32},
- thin_mode: {type: Bool, default: false},
- unseen_only: {type: Bool, default: false},
- video_loop: {type: Bool, default: false},
- volume: {type: Int32, default: 100},
- })
end
struct Config
- module ConfigPreferencesConverter
- def self.to_yaml(value : Preferences, yaml : YAML::Nodes::Builder)
- value.to_yaml(yaml)
- end
-
- def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : Preferences
- Preferences.new(*ConfigPreferences.new(ctx, node).to_tuple)
- end
- end
-
- module FamilyConverter
- def self.to_yaml(value : Socket::Family, yaml : YAML::Nodes::Builder)
- case value
- when Socket::Family::UNSPEC
- yaml.scalar nil
- when Socket::Family::INET
- yaml.scalar "ipv4"
- when Socket::Family::INET6
- yaml.scalar "ipv6"
- when Socket::Family::UNIX
- raise "Invalid socket family #{value}"
- end
- end
-
- def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : Socket::Family
- if node.is_a?(YAML::Nodes::Scalar)
- case node.value.downcase
- when "ipv4"
- Socket::Family::INET
- when "ipv6"
- Socket::Family::INET6
- else
- Socket::Family::UNSPEC
- end
- else
- node.raise "Expected scalar, not #{node.class}"
- end
- end
- end
-
- module StringToCookies
- def self.to_yaml(value : HTTP::Cookies, yaml : YAML::Nodes::Builder)
- (value.map { |c| "#{c.name}=#{c.value}" }).join("; ").to_yaml(yaml)
- end
-
- def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : HTTP::Cookies
- unless node.is_a?(YAML::Nodes::Scalar)
- node.raise "Expected scalar, not #{node.class}"
- end
-
- cookies = HTTP::Cookies.new
- node.value.split(";").each do |cookie|
- next if cookie.strip.empty?
- name, value = cookie.split("=", 2)
- cookies << HTTP::Cookie.new(name.strip, value.strip)
- end
-
- cookies
- end
- end
+ include YAML::Serializable
+
+ property channel_threads : Int32 # Number of threads to use for crawling videos from channels (for updating subscriptions)
+ property feed_threads : Int32 # Number of threads to use for updating feeds
+ property db : DBConfig # Database configuration
+ property full_refresh : Bool # Used for crawling channels: threads should check all videos uploaded by a channel
+ property https_only : Bool? # Used to tell Invidious it is behind a proxy, so links to resources should be https://
+ property hmac_key : String? # HMAC signing key for CSRF tokens and verifying pubsub subscriptions
+ property domain : String? # Domain to be used for links to resources on the site where an absolute URL is required
+ property use_pubsub_feeds : Bool | Int32 = false # Subscribe to channels using PubSubHubbub (requires domain, hmac_key)
+ property captcha_enabled : Bool = true
+ property login_enabled : Bool = true
+ property registration_enabled : Bool = true
+ property statistics_enabled : Bool = false
+ property admins : Array(String) = [] of String
+ property external_port : Int32? = nil
+ property default_user_preferences : ConfigPreferences = ConfigPreferences.from_yaml("")
+ property dmca_content : Array(String) = [] of String # For compliance with DMCA, disables download widget using list of video IDs
+ property check_tables : Bool = false # Check table integrity, automatically try to add any missing columns, create tables, etc.
+ property cache_annotations : Bool = false # Cache annotations requested from IA, will not cache empty annotations or annotations that only contain cards
+ property banner : String? = nil # Optional banner to be displayed along top of page for announcements, etc.
+ property hsts : Bool? = true # Enables 'Strict-Transport-Security'. Ensure that `domain` and all subdomains are served securely
+ property disable_proxy : Bool? | Array(String)? = false # Disable proxying server-wide: options: 'dash', 'livestreams', 'downloads', 'local'
+
+ @[YAML::Field(converter: Preferences::FamilyConverter)]
+ property force_resolve : Socket::Family = Socket::Family::UNSPEC # Connect to YouTube over 'ipv6', 'ipv4'. Will sometimes resolve fix issues with rate-limiting (see https://github.com/ytdl-org/youtube-dl/issues/21729)
+ property port : Int32 = 3000 # Port to listen for connections (overrided by command line argument)
+ property host_binding : String = "0.0.0.0" # Host to bind (overrided by command line argument)
+ property pool_size : Int32 = 100 # Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
+ property admin_email : String = "omarroth@protonmail.com" # Email for bug reports
+
+ @[YAML::Field(converter: Preferences::StringToCookies)]
+ property cookies : HTTP::Cookies = HTTP::Cookies.new # Saved cookies in "name1=value1; name2=value2..." format
+ property captcha_key : String? = nil # Key for Anti-Captcha
def disabled?(option)
case disabled = CONFIG.disable_proxy
@@ -229,50 +110,16 @@ struct Config
return false
end
end
-
- YAML.mapping({
- channel_threads: Int32, # Number of threads to use for crawling videos from channels (for updating subscriptions)
- feed_threads: Int32, # Number of threads to use for updating feeds
- db: DBConfig, # Database configuration
- full_refresh: Bool, # Used for crawling channels: threads should check all videos uploaded by a channel
- https_only: Bool?, # Used to tell Invidious it is behind a proxy, so links to resources should be https://
- hmac_key: String?, # HMAC signing key for CSRF tokens and verifying pubsub subscriptions
- domain: String?, # Domain to be used for links to resources on the site where an absolute URL is required
- use_pubsub_feeds: {type: Bool | Int32, default: false}, # Subscribe to channels using PubSubHubbub (requires domain, hmac_key)
- captcha_enabled: {type: Bool, default: true},
- login_enabled: {type: Bool, default: true},
- registration_enabled: {type: Bool, default: true},
- statistics_enabled: {type: Bool, default: false},
- admins: {type: Array(String), default: [] of String},
- external_port: {type: Int32?, default: nil},
- default_user_preferences: {type: Preferences,
- default: Preferences.new(*ConfigPreferences.from_yaml("").to_tuple),
- converter: ConfigPreferencesConverter,
- },
- dmca_content: {type: Array(String), default: [] of String}, # For compliance with DMCA, disables download widget using list of video IDs
- check_tables: {type: Bool, default: false}, # Check table integrity, automatically try to add any missing columns, create tables, etc.
- cache_annotations: {type: Bool, default: false}, # Cache annotations requested from IA, will not cache empty annotations or annotations that only contain cards
- banner: {type: String?, default: nil}, # Optional banner to be displayed along top of page for announcements, etc.
- hsts: {type: Bool?, default: true}, # Enables 'Strict-Transport-Security'. Ensure that `domain` and all subdomains are served securely
- disable_proxy: {type: Bool? | Array(String)?, default: false}, # Disable proxying server-wide: options: 'dash', 'livestreams', 'downloads', 'local'
- force_resolve: {type: Socket::Family, default: Socket::Family::UNSPEC, converter: FamilyConverter}, # Connect to YouTube over 'ipv6', 'ipv4'. Will sometimes resolve fix issues with rate-limiting (see https://github.com/ytdl-org/youtube-dl/issues/21729)
- port: {type: Int32, default: 3000}, # Port to listen for connections (overrided by command line argument)
- host_binding: {type: String, default: "0.0.0.0"}, # Host to bind (overrided by command line argument)
- pool_size: {type: Int32, default: 100}, # Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
- admin_email: {type: String, default: "omarroth@protonmail.com"}, # Email for bug reports
- cookies: {type: HTTP::Cookies, default: HTTP::Cookies.new, converter: StringToCookies}, # Saved cookies in "name1=value1; name2=value2..." format
- captcha_key: {type: String?, default: nil}, # Key for Anti-Captcha
- })
end
struct DBConfig
- yaml_mapping({
- user: String,
- password: String,
- host: String,
- port: Int32,
- dbname: String,
- })
+ include YAML::Serializable
+
+ property user : String
+ property password : String
+ property host : String
+ property port : Int32
+ property dbname : String
end
def login_req(f_req)
@@ -326,7 +173,7 @@ def extract_items(initial_data : Hash(String, JSON::Any), author_fallback : Stri
t["continuationContents"]? }
.try { |t| t["sectionListRenderer"]? || t["sectionListContinuation"]? }
.try &.["contents"].as_a
- .each { |c| c.try &.["itemSectionRenderer"]["contents"].as_a
+ .each { |c| c.try &.["itemSectionRenderer"]?.try &.["contents"].as_a
.try { |t| t[0]?.try &.["shelfRenderer"]?.try &.["content"]["expandedShelfContentsRenderer"]?.try &.["items"].as_a ||
t[0]?.try &.["gridRenderer"]?.try &.["items"].as_a || t }
.each { |item|
@@ -365,20 +212,20 @@ def extract_items(initial_data : Hash(String, JSON::Any), author_fallback : Stri
end
end
- items << SearchVideo.new(
- title: title,
- id: video_id,
- author: author,
- ucid: author_id,
- published: published,
- views: view_count,
- description_html: description_html,
- length_seconds: length_seconds,
- live_now: live_now,
- paid: paid,
- premium: premium,
- premiere_timestamp: premiere_timestamp
- )
+ items << SearchVideo.new({
+ title: title,
+ id: video_id,
+ author: author,
+ ucid: author_id,
+ published: published,
+ views: view_count,
+ description_html: description_html,
+ length_seconds: length_seconds,
+ live_now: live_now,
+ paid: paid,
+ premium: premium,
+ premiere_timestamp: premiere_timestamp,
+ })
elsif i = item["channelRenderer"]?
author = i["title"]["simpleText"]?.try &.as_s || author_fallback || ""
author_id = i["channelId"]?.try &.as_s || author_id_fallback || ""
@@ -391,15 +238,31 @@ def extract_items(initial_data : Hash(String, JSON::Any), author_fallback : Stri
video_count = i["videoCountText"]?.try &.["runs"].as_a[0]?.try &.["text"].as_s.gsub(/\D/, "").to_i || 0
description_html = i["descriptionSnippet"]?.try { |t| parse_content(t) } || ""
- items << SearchChannel.new(
- author: author,
- ucid: author_id,
+ items << SearchChannel.new({
+ author: author,
+ ucid: author_id,
author_thumbnail: author_thumbnail,
subscriber_count: subscriber_count,
- video_count: video_count,
+ video_count: video_count,
description_html: description_html,
- auto_generated: auto_generated,
- )
+ auto_generated: auto_generated,
+ })
+ elsif i = item["gridPlaylistRenderer"]?
+ title = i["title"]["runs"].as_a[0]?.try &.["text"].as_s || ""
+ plid = i["playlistId"]?.try &.as_s || ""
+
+ video_count = i["videoCountText"]["runs"].as_a[0]?.try &.["text"].as_s.gsub(/\D/, "").to_i || 0
+ playlist_thumbnail = i["thumbnail"]["thumbnails"][0]?.try &.["url"]?.try &.as_s || ""
+
+ items << SearchPlaylist.new({
+ title: title,
+ id: plid,
+ author: author_fallback || "",
+ ucid: author_id_fallback || "",
+ video_count: video_count,
+ videos: [] of SearchPlaylistVideo,
+ thumbnail: playlist_thumbnail,
+ })
elsif i = item["playlistRenderer"]?
title = i["title"]["simpleText"]?.try &.as_s || ""
plid = i["playlistId"]?.try &.as_s || ""
@@ -416,24 +279,24 @@ def extract_items(initial_data : Hash(String, JSON::Any), author_fallback : Stri
v_title = v["title"]["simpleText"]?.try &.as_s || ""
v_id = v["videoId"]?.try &.as_s || ""
v_length_seconds = v["lengthText"]?.try &.["simpleText"]?.try { |t| decode_length_seconds(t.as_s) } || 0
- SearchPlaylistVideo.new(
- title: v_title,
- id: v_id,
- length_seconds: v_length_seconds
- )
+ SearchPlaylistVideo.new({
+ title: v_title,
+ id: v_id,
+ length_seconds: v_length_seconds,
+ })
end || [] of SearchPlaylistVideo
# TODO: i["publishedTimeText"]?
- items << SearchPlaylist.new(
- title: title,
- id: plid,
- author: author,
- ucid: author_id,
+ items << SearchPlaylist.new({
+ title: title,
+ id: plid,
+ author: author,
+ ucid: author_id,
video_count: video_count,
- videos: videos,
- thumbnail: playlist_thumbnail
- )
+ videos: videos,
+ thumbnail: playlist_thumbnail,
+ })
elsif i = item["radioRenderer"]? # Mix
# TODO
elsif i = item["showRenderer"]? # Show
@@ -449,6 +312,7 @@ end
def check_enum(db, logger, enum_name, struct_type = nil)
return # TODO
+
if !db.query_one?("SELECT true FROM pg_type WHERE typname = $1", enum_name, as: Bool)
logger.puts("CREATE TYPE #{enum_name}")
@@ -472,7 +336,7 @@ def check_table(db, logger, table_name, struct_type = nil)
return if !struct_type
- struct_array = struct_type.to_type_tuple
+ struct_array = struct_type.type_array
column_array = get_column_array(db, table_name)
column_types = File.read("config/sql/#{table_name}.sql").match(/CREATE TABLE public\.#{table_name}\n\((?<types>[\d\D]*?)\);/)
.try &.["types"].split(",").map { |line| line.strip }.reject &.starts_with?("CONSTRAINT")
diff --git a/src/invidious/helpers/jobs.cr b/src/invidious/helpers/jobs.cr
index befed471..4594c1e0 100644
--- a/src/invidious/helpers/jobs.cr
+++ b/src/invidious/helpers/jobs.cr
@@ -67,7 +67,7 @@ def refresh_feeds(db, logger, config)
begin
# Drop outdated views
column_array = get_column_array(db, view_name)
- ChannelVideo.to_type_tuple.each_with_index do |name, i|
+ ChannelVideo.type_array.each_with_index do |name, i|
if name != column_array[i]?
logger.puts("DROP MATERIALIZED VIEW #{view_name}")
db.exec("DROP MATERIALIZED VIEW #{view_name}")
diff --git a/src/invidious/helpers/macros.cr b/src/invidious/helpers/macros.cr
index ddfb9f8e..8b74bc86 100644
--- a/src/invidious/helpers/macros.cr
+++ b/src/invidious/helpers/macros.cr
@@ -1,43 +1,51 @@
-macro db_mapping(mapping)
- def initialize({{*mapping.keys.map { |id| "@#{id}".id }}})
+module DB::Serializable
+ macro included
+ {% verbatim do %}
+ macro finished
+ def self.type_array
+ \{{ @type.instance_vars
+ .reject { |var| var.annotation(::DB::Field) && var.annotation(::DB::Field)[:ignore] }
+ .map { |name| name.stringify }
+ }}
+ end
+
+ def initialize(tuple)
+ \{% for var in @type.instance_vars %}
+ \{% ann = var.annotation(::DB::Field) %}
+ \{% if ann && ann[:ignore] %}
+ \{% else %}
+ @\{{var.name}} = tuple[:\{{var.name.id}}]
+ \{% end %}
+ \{% end %}
+ end
+
+ def to_a
+ \{{ @type.instance_vars
+ .reject { |var| var.annotation(::DB::Field) && var.annotation(::DB::Field)[:ignore] }
+ .map { |name| name }
+ }}
+ end
+ end
+ {% end %}
end
-
- def to_a
- return [ {{*mapping.keys.map { |id| "@#{id}".id }}} ]
- end
-
- def self.to_type_tuple
- return { {{*mapping.keys.map { |id| "#{id}" }}} }
- end
-
- DB.mapping( {{mapping}} )
-end
-
-macro json_mapping(mapping)
- def initialize({{*mapping.keys.map { |id| "@#{id}".id }}})
- end
-
- def to_a
- return [ {{*mapping.keys.map { |id| "@#{id}".id }}} ]
- end
-
- patched_json_mapping( {{mapping}} )
- YAML.mapping( {{mapping}} )
end
-macro yaml_mapping(mapping)
- def initialize({{*mapping.keys.map { |id| "@#{id}".id }}})
- end
-
- def to_a
- return [ {{*mapping.keys.map { |id| "@#{id}".id }}} ]
- end
-
- def to_tuple
- return { {{*mapping.keys.map { |id| "@#{id}".id }}} }
+module JSON::Serializable
+ macro included
+ {% verbatim do %}
+ macro finished
+ def initialize(tuple)
+ \{% for var in @type.instance_vars %}
+ \{% ann = var.annotation(::JSON::Field) %}
+ \{% if ann && ann[:ignore] %}
+ \{% else %}
+ @\{{var.name}} = tuple[:\{{var.name.id}}]
+ \{% end %}
+ \{% end %}
+ end
+ end
+ {% end %}
end
-
- YAML.mapping({{mapping}})
end
macro templated(filename, template = "template")
diff --git a/src/invidious/helpers/patch_mapping.cr b/src/invidious/helpers/patch_mapping.cr
deleted file mode 100644
index 19bd8ca1..00000000
--- a/src/invidious/helpers/patch_mapping.cr
+++ /dev/null
@@ -1,166 +0,0 @@
-# Overloads https://github.com/crystal-lang/crystal/blob/0.28.0/src/json/from_json.cr#L24
-def Object.from_json(string_or_io, default) : self
- parser = JSON::PullParser.new(string_or_io)
- new parser, default
-end
-
-# Adds configurable 'default'
-macro patched_json_mapping(_properties_, strict = false)
- {% for key, value in _properties_ %}
- {% _properties_[key] = {type: value} unless value.is_a?(HashLiteral) || value.is_a?(NamedTupleLiteral) %}
- {% end %}
-
- {% for key, value in _properties_ %}
- {% _properties_[key][:key_id] = key.id.gsub(/\?$/, "") %}
- {% end %}
-
- {% for key, value in _properties_ %}
- @{{value[:key_id]}} : {{value[:type]}}{{ (value[:nilable] ? "?" : "").id }}
-
- {% if value[:setter] == nil ? true : value[:setter] %}
- def {{value[:key_id]}}=(_{{value[:key_id]}} : {{value[:type]}}{{ (value[:nilable] ? "?" : "").id }})
- @{{value[:key_id]}} = _{{value[:key_id]}}
- end
- {% end %}
-
- {% if value[:getter] == nil ? true : value[:getter] %}
- def {{key.id}} : {{value[:type]}}{{ (value[:nilable] ? "?" : "").id }}
- @{{value[:key_id]}}
- end
- {% end %}
-
- {% if value[:presence] %}
- @{{value[:key_id]}}_present : Bool = false
-
- def {{value[:key_id]}}_present?
- @{{value[:key_id]}}_present
- end
- {% end %}
- {% end %}
-
- def initialize(%pull : ::JSON::PullParser, default = nil)
- {% for key, value in _properties_ %}
- %var{key.id} = nil
- %found{key.id} = false
- {% end %}
-
- %location = %pull.location
- begin
- %pull.read_begin_object
- rescue exc : ::JSON::ParseException
- raise ::JSON::MappingError.new(exc.message, self.class.to_s, nil, *%location, exc)
- end
- until %pull.kind.end_object?
- %key_location = %pull.location
- key = %pull.read_object_key
- case key
- {% for key, value in _properties_ %}
- when {{value[:key] || value[:key_id].stringify}}
- %found{key.id} = true
- begin
- %var{key.id} =
- {% if value[:nilable] || value[:default] != nil %} %pull.read_null_or { {% end %}
-
- {% if value[:root] %}
- %pull.on_key!({{value[:root]}}) do
- {% end %}
-
- {% if value[:converter] %}
- {{value[:converter]}}.from_json(%pull)
- {% elsif value[:type].is_a?(Path) || value[:type].is_a?(Generic) %}
- {{value[:type]}}.new(%pull)
- {% else %}
- ::Union({{value[:type]}}).new(%pull)
- {% end %}
-
- {% if value[:root] %}
- end
- {% end %}
-
- {% if value[:nilable] || value[:default] != nil %} } {% end %}
- rescue exc : ::JSON::ParseException
- raise ::JSON::MappingError.new(exc.message, self.class.to_s, {{value[:key] || value[:key_id].stringify}}, *%key_location, exc)
- end
- {% end %}
- else
- {% if strict %}
- raise ::JSON::MappingError.new("Unknown JSON attribute: #{key}", self.class.to_s, nil, *%key_location, nil)
- {% else %}
- %pull.skip
- {% end %}
- end
- end
- %pull.read_next
-
- {% for key, value in _properties_ %}
- {% unless value[:nilable] || value[:default] != nil %}
- if %var{key.id}.nil? && !%found{key.id} && !::Union({{value[:type]}}).nilable?
- raise ::JSON::MappingError.new("Missing JSON attribute: {{(value[:key] || value[:key_id]).id}}", self.class.to_s, nil, *%location, nil)
- end
- {% end %}
-
- {% if value[:nilable] %}
- {% if value[:default] != nil %}
- @{{value[:key_id]}} = %found{key.id} ? %var{key.id} : (default.responds_to?(:{{value[:key_id]}}) ? default.{{value[:key_id]}} : {{value[:default]}})
- {% else %}
- @{{value[:key_id]}} = %var{key.id}
- {% end %}
- {% elsif value[:default] != nil %}
- @{{value[:key_id]}} = %var{key.id}.nil? ? (default.responds_to?(:{{value[:key_id]}}) ? default.{{value[:key_id]}} : {{value[:default]}}) : %var{key.id}
- {% else %}
- @{{value[:key_id]}} = (%var{key.id}).as({{value[:type]}})
- {% end %}
-
- {% if value[:presence] %}
- @{{value[:key_id]}}_present = %found{key.id}
- {% end %}
- {% end %}
- end
-
- def to_json(json : ::JSON::Builder)
- json.object do
- {% for key, value in _properties_ %}
- _{{value[:key_id]}} = @{{value[:key_id]}}
-
- {% unless value[:emit_null] %}
- unless _{{value[:key_id]}}.nil?
- {% end %}
-
- json.field({{value[:key] || value[:key_id].stringify}}) do
- {% if value[:root] %}
- {% if value[:emit_null] %}
- if _{{value[:key_id]}}.nil?
- nil.to_json(json)
- else
- {% end %}
-
- json.object do
- json.field({{value[:root]}}) do
- {% end %}
-
- {% if value[:converter] %}
- if _{{value[:key_id]}}
- {{ value[:converter] }}.to_json(_{{value[:key_id]}}, json)
- else
- nil.to_json(json)
- end
- {% else %}
- _{{value[:key_id]}}.to_json(json)
- {% end %}
-
- {% if value[:root] %}
- {% if value[:emit_null] %}
- end
- {% end %}
- end
- end
- {% end %}
- end
-
- {% unless value[:emit_null] %}
- end
- {% end %}
- {% end %}
- end
- end
-end
diff --git a/src/invidious/mixes.cr b/src/invidious/mixes.cr
index 6c01d78b..c69eb0c4 100644
--- a/src/invidious/mixes.cr
+++ b/src/invidious/mixes.cr
@@ -1,21 +1,21 @@
struct MixVideo
- db_mapping({
- title: String,
- id: String,
- author: String,
- ucid: String,
- length_seconds: Int32,
- index: Int32,
- rdid: String,
- })
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property ucid : String
+ property length_seconds : Int32
+ property index : Int32
+ property rdid : String
end
struct Mix
- db_mapping({
- title: String,
- id: String,
- videos: Array(MixVideo),
- })
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property videos : Array(MixVideo)
end
def fetch_mix(rdid, video_id, cookies = nil, locale = nil)
@@ -24,8 +24,9 @@ def fetch_mix(rdid, video_id, cookies = nil, locale = nil)
if cookies
headers = cookies.add_request_headers(headers)
end
- response = YT_POOL.client &.get("/watch?v=#{video_id}&list=#{rdid}&gl=US&hl=en&has_verified=1&bpctr=9999999999", headers)
+ video_id = "CvFH_6DNRCY" if rdid.starts_with? "OLAK5uy_"
+ response = YT_POOL.client &.get("/watch?v=#{video_id}&list=#{rdid}&gl=US&hl=en", headers)
initial_data = extract_initial_data(response.body)
if !initial_data["contents"]["twoColumnWatchNextResults"]["playlist"]?
@@ -48,23 +49,22 @@ def fetch_mix(rdid, video_id, cookies = nil, locale = nil)
id = item["videoId"].as_s
title = item["title"]?.try &.["simpleText"].as_s
- if !title
- next
- end
+ next if !title
+
author = item["longBylineText"]["runs"][0]["text"].as_s
ucid = item["longBylineText"]["runs"][0]["navigationEndpoint"]["browseEndpoint"]["browseId"].as_s
length_seconds = decode_length_seconds(item["lengthText"]["simpleText"].as_s)
index = item["navigationEndpoint"]["watchEndpoint"]["index"].as_i
- videos << MixVideo.new(
- title,
- id,
- author,
- ucid,
- length_seconds,
- index,
- rdid
- )
+ videos << MixVideo.new({
+ title: title,
+ id: id,
+ author: author,
+ ucid: ucid,
+ length_seconds: length_seconds,
+ index: index,
+ rdid: rdid,
+ })
end
if !cookies
@@ -74,7 +74,11 @@ def fetch_mix(rdid, video_id, cookies = nil, locale = nil)
videos.uniq! { |video| video.id }
videos = videos.first(50)
- return Mix.new(mix_title, rdid, videos)
+ return Mix.new({
+ title: mix_title,
+ id: rdid,
+ videos: videos,
+ })
end
def template_mix(mix)
diff --git a/src/invidious/playlists.cr b/src/invidious/playlists.cr
index fcf73dad..9190e4e6 100644
--- a/src/invidious/playlists.cr
+++ b/src/invidious/playlists.cr
@@ -1,4 +1,16 @@
struct PlaylistVideo
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property ucid : String
+ property length_seconds : Int32
+ property published : Time
+ property plid : String
+ property index : Int64
+ property live_now : Bool
+
def to_xml(auto_generated, xml : XML::Builder)
xml.element("entry") do
xml.element("id") { xml.text "yt:video:#{self.id}" }
@@ -78,21 +90,22 @@ struct PlaylistVideo
end
end
end
-
- db_mapping({
- title: String,
- id: String,
- author: String,
- ucid: String,
- length_seconds: Int32,
- published: Time,
- plid: String,
- index: Int64,
- live_now: Bool,
- })
end
struct Playlist
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property author_thumbnail : String
+ property ucid : String
+ property description : String
+ property video_count : Int32
+ property views : Int64
+ property updated : Time
+ property thumbnail : String?
+
def to_json(offset, locale, json : JSON::Builder, continuation : String? = nil)
json.object do
json.field "type", "playlist"
@@ -118,7 +131,7 @@ struct Playlist
end
end
- json.field "description", html_to_content(self.description_html)
+ json.field "description", self.description
json.field "descriptionHtml", self.description_html
json.field "videoCount", self.video_count
@@ -147,22 +160,13 @@ struct Playlist
end
end
- db_mapping({
- title: String,
- id: String,
- author: String,
- author_thumbnail: String,
- ucid: String,
- description_html: String,
- video_count: Int32,
- views: Int64,
- updated: Time,
- thumbnail: String?,
- })
-
def privacy
PlaylistPrivacy::Public
end
+
+ def description_html
+ HTML.escape(self.description).gsub("\n", "<br>")
+ end
end
enum PlaylistPrivacy
@@ -172,6 +176,29 @@ enum PlaylistPrivacy
end
struct InvidiousPlaylist
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property description : String = ""
+ property video_count : Int32
+ property created : Time
+ property updated : Time
+
+ @[DB::Field(converter: InvidiousPlaylist::PlaylistPrivacyConverter)]
+ property privacy : PlaylistPrivacy = PlaylistPrivacy::Private
+ property index : Array(Int64)
+
+ @[DB::Field(ignore: true)]
+ property thumbnail_id : String?
+
+ module PlaylistPrivacyConverter
+ def self.from_rs(rs)
+ return PlaylistPrivacy.parse(String.new(rs.read(Slice(UInt8))))
+ end
+ end
+
def to_json(offset, locale, json : JSON::Builder, continuation : String? = nil)
json.object do
json.field "type", "invidiousPlaylist"
@@ -212,26 +239,6 @@ struct InvidiousPlaylist
end
end
- property thumbnail_id
-
- module PlaylistPrivacyConverter
- def self.from_rs(rs)
- return PlaylistPrivacy.parse(String.new(rs.read(Slice(UInt8))))
- end
- end
-
- db_mapping({
- title: String,
- id: String,
- author: String,
- description: {type: String, default: ""},
- video_count: Int32,
- created: Time,
- updated: Time,
- privacy: {type: PlaylistPrivacy, default: PlaylistPrivacy::Private, converter: PlaylistPrivacyConverter},
- index: Array(Int64),
- })
-
def thumbnail
@thumbnail_id ||= PG_DB.query_one?("SELECT id FROM playlist_videos WHERE plid = $1 ORDER BY array_position($2, index) LIMIT 1", self.id, self.index, as: String) || "-----------"
"/vi/#{@thumbnail_id}/mqdefault.jpg"
@@ -257,17 +264,17 @@ end
def create_playlist(db, title, privacy, user)
plid = "IVPL#{Random::Secure.urlsafe_base64(24)[0, 31]}"
- playlist = InvidiousPlaylist.new(
- title: title.byte_slice(0, 150),
- id: plid,
- author: user.email,
+ playlist = InvidiousPlaylist.new({
+ title: title.byte_slice(0, 150),
+ id: plid,
+ author: user.email,
description: "", # Max 5000 characters
video_count: 0,
- created: Time.utc,
- updated: Time.utc,
- privacy: privacy,
- index: [] of Int64,
- )
+ created: Time.utc,
+ updated: Time.utc,
+ privacy: privacy,
+ index: [] of Int64,
+ })
playlist_array = playlist.to_a
args = arg_array(playlist_array)
@@ -278,17 +285,17 @@ def create_playlist(db, title, privacy, user)
end
def subscribe_playlist(db, user, playlist)
- playlist = InvidiousPlaylist.new(
- title: playlist.title.byte_slice(0, 150),
- id: playlist.id,
- author: user.email,
+ playlist = InvidiousPlaylist.new({
+ title: playlist.title.byte_slice(0, 150),
+ id: playlist.id,
+ author: user.email,
description: "", # Max 5000 characters
video_count: playlist.video_count,
- created: Time.utc,
- updated: playlist.updated,
- privacy: PlaylistPrivacy::Private,
- index: [] of Int64,
- )
+ created: Time.utc,
+ updated: playlist.updated,
+ privacy: PlaylistPrivacy::Private,
+ index: [] of Int64,
+ })
playlist_array = playlist.to_a
args = arg_array(playlist_array)
@@ -298,52 +305,6 @@ def subscribe_playlist(db, user, playlist)
return playlist
end
-def extract_playlist(plid, nodeset, index)
- videos = [] of PlaylistVideo
-
- nodeset.each_with_index do |video, offset|
- anchor = video.xpath_node(%q(.//td[@class="pl-video-title"]))
- if !anchor
- next
- end
-
- title = anchor.xpath_node(%q(.//a)).not_nil!.content.strip(" \n")
- id = anchor.xpath_node(%q(.//a)).not_nil!["href"].lchop("/watch?v=")[0, 11]
-
- anchor = anchor.xpath_node(%q(.//div[@class="pl-video-owner"]/a))
- if anchor
- author = anchor.content
- ucid = anchor["href"].split("/")[2]
- else
- author = ""
- ucid = ""
- end
-
- anchor = video.xpath_node(%q(.//td[@class="pl-video-time"]/div/div[1]))
- if anchor && !anchor.content.empty?
- length_seconds = decode_length_seconds(anchor.content)
- live_now = false
- else
- length_seconds = 0
- live_now = true
- end
-
- videos << PlaylistVideo.new(
- title: title,
- id: id,
- author: author,
- ucid: ucid,
- length_seconds: length_seconds,
- published: Time.utc,
- plid: plid,
- index: (index + offset).to_i64,
- live_now: live_now
- )
- end
-
- return videos
-end
-
def produce_playlist_url(id, index)
if id.starts_with? "UC"
id = "UU" + id.lchop("UC")
@@ -389,58 +350,64 @@ def fetch_playlist(plid, locale)
plid = "UU#{plid.lchop("UC")}"
end
- response = YT_POOL.client &.get("/playlist?list=#{plid}&hl=en&disable_polymer=1")
+ response = YT_POOL.client &.get("/playlist?list=#{plid}&hl=en")
if response.status_code != 200
- raise translate(locale, "Not a playlist.")
+ if response.headers["location"]?.try &.includes? "/sorry/index"
+ raise "Could not extract playlist info. Instance is likely blocked."
+ else
+ raise translate(locale, "Not a playlist.")
+ end
end
- body = response.body.gsub(/<button[^>]+><span[^>]+>\s*less\s*<img[^>]+>\n<\/span><\/button>/, "")
- document = XML.parse_html(body)
+ initial_data = extract_initial_data(response.body)
+ playlist_info = initial_data["sidebar"]?.try &.["playlistSidebarRenderer"]?.try &.["items"]?.try &.[0]["playlistSidebarPrimaryInfoRenderer"]?
- title = document.xpath_node(%q(//h1[@class="pl-header-title"]))
- if !title
- raise translate(locale, "Playlist does not exist.")
- end
- title = title.content.strip(" \n")
+ raise "Could not extract playlist info" if !playlist_info
+ title = playlist_info["title"]?.try &.["runs"][0]?.try &.["text"]?.try &.as_s || ""
- description_html = document.xpath_node(%q(//span[@class="pl-header-description-text"]/div/div[1])).try &.to_s ||
- document.xpath_node(%q(//span[@class="pl-header-description-text"])).try &.to_s || ""
+ desc_item = playlist_info["description"]?
+ description = desc_item.try &.["runs"]?.try &.as_a.map(&.["text"].as_s).join("") || desc_item.try &.["simpleText"]?.try &.as_s || ""
- playlist_thumbnail = document.xpath_node(%q(//div[@class="pl-header-thumb"]/img)).try &.["data-thumb"]? ||
- document.xpath_node(%q(//div[@class="pl-header-thumb"]/img)).try &.["src"]
+ thumbnail = playlist_info["thumbnailRenderer"]?.try &.["playlistVideoThumbnailRenderer"]?
+ .try &.["thumbnail"]["thumbnails"][0]["url"]?.try &.as_s
- # YouTube allows anonymous playlists, so most of this can be empty or optional
- anchor = document.xpath_node(%q(//ul[@class="pl-header-details"]))
- author = anchor.try &.xpath_node(%q(.//li[1]/a)).try &.content
- author ||= ""
- author_thumbnail = document.xpath_node(%q(//img[@class="channel-header-profile-image"])).try &.["src"]
- author_thumbnail ||= ""
- ucid = anchor.try &.xpath_node(%q(.//li[1]/a)).try &.["href"].split("/")[-1]
- ucid ||= ""
+ views = 0_i64
+ updated = Time.utc
+ video_count = 0
+ playlist_info["stats"]?.try &.as_a.each do |stat|
+ text = stat["runs"]?.try &.as_a.map(&.["text"].as_s).join("") || stat["simpleText"]?.try &.as_s
+ next if !text
- video_count = anchor.try &.xpath_node(%q(.//li[2])).try &.content.gsub(/\D/, "").to_i?
- video_count ||= 0
+ if text.includes? "video"
+ video_count = text.gsub(/\D/, "").to_i? || 0
+ elsif text.includes? "view"
+ views = text.gsub(/\D/, "").to_i64? || 0_i64
+ else
+ updated = decode_date(text.lchop("Last updated on ").lchop("Updated "))
+ end
+ end
- views = anchor.try &.xpath_node(%q(.//li[3])).try &.content.gsub(/\D/, "").to_i64?
- views ||= 0_i64
+ author_info = initial_data["sidebar"]?.try &.["playlistSidebarRenderer"]?.try &.["items"]?.try &.[1]["playlistSidebarSecondaryInfoRenderer"]?
+ .try &.["videoOwner"]["videoOwnerRenderer"]?
- updated = anchor.try &.xpath_node(%q(.//li[4])).try &.content.lchop("Last updated on ").lchop("Updated ").try { |date| decode_date(date) }
- updated ||= Time.utc
+ raise "Could not extract author info" if !author_info
- playlist = Playlist.new(
- title: title,
- id: plid,
- author: author,
- author_thumbnail: author_thumbnail,
- ucid: ucid,
- description_html: description_html,
- video_count: video_count,
- views: views,
- updated: updated,
- thumbnail: playlist_thumbnail,
- )
+ author_thumbnail = author_info["thumbnail"]["thumbnails"][0]["url"]?.try &.as_s || ""
+ author = author_info["title"]["runs"][0]["text"]?.try &.as_s || ""
+ ucid = author_info["title"]["runs"][0]["navigationEndpoint"]["browseEndpoint"]["browseId"]?.try &.as_s || ""
- return playlist
+ return Playlist.new({
+ title: title,
+ id: plid,
+ author: author,
+ author_thumbnail: author_thumbnail,
+ ucid: ucid,
+ description: description,
+ video_count: video_count,
+ views: views,
+ updated: updated,
+ thumbnail: thumbnail,
+ })
end
def get_playlist_videos(db, playlist, offset, locale = nil, continuation = nil)
@@ -458,35 +425,26 @@ end
def fetch_playlist_videos(plid, video_count, offset = 0, locale = nil, continuation = nil)
if continuation
- html = YT_POOL.client &.get("/watch?v=#{continuation}&list=#{plid}&gl=US&hl=en&disable_polymer=1&has_verified=1&bpctr=9999999999")
- html = XML.parse_html(html.body)
-
- index = html.xpath_node(%q(//span[@id="playlist-current-index"])).try &.content.to_i?.try &.- 1
- offset = index || offset
+ response = YT_POOL.client &.get("/watch?v=#{continuation}&list=#{plid}&gl=US&hl=en")
+ initial_data = extract_initial_data(response.body)
+ offset = initial_data["currentVideoEndpoint"]?.try &.["watchEndpoint"]?.try &.["index"]?.try &.as_i64 || offset
end
if video_count > 100
url = produce_playlist_url(plid, offset)
response = YT_POOL.client &.get(url)
- response = JSON.parse(response.body)
- if !response["content_html"]? || response["content_html"].as_s.empty?
- raise translate(locale, "Empty playlist")
- end
-
- document = XML.parse_html(response["content_html"].as_s)
- nodeset = document.xpath_nodes(%q(.//tr[contains(@class, "pl-video")]))
- videos = extract_playlist(plid, nodeset, offset)
+ initial_data = JSON.parse(response.body).as_a.find(&.as_h.["response"]?).try &.as_h
elsif offset > 100
return [] of PlaylistVideo
else # Extract first page of videos
- response = YT_POOL.client &.get("/playlist?list=#{plid}&gl=US&hl=en&disable_polymer=1")
- document = XML.parse_html(response.body)
- nodeset = document.xpath_nodes(%q(.//tr[contains(@class, "pl-video")]))
-
- videos = extract_playlist(plid, nodeset, 0)
+ response = YT_POOL.client &.get("/playlist?list=#{plid}&gl=US&hl=en")
+ initial_data = extract_initial_data(response.body)
end
+ return [] of PlaylistVideo if !initial_data
+ videos = extract_playlist_videos(initial_data)
+
until videos.empty? || videos[0].index == offset
videos.shift
end
@@ -494,6 +452,45 @@ def fetch_playlist_videos(plid, video_count, offset = 0, locale = nil, continuat
return videos
end
+def extract_playlist_videos(initial_data : Hash(String, JSON::Any))
+ videos = [] of PlaylistVideo
+
+ (initial_data["contents"]?.try &.["twoColumnBrowseResultsRenderer"]["tabs"].as_a.select(&.["tabRenderer"]["selected"]?.try &.as_bool)[0]["tabRenderer"]["content"]["sectionListRenderer"]["contents"][0]["itemSectionRenderer"]["contents"][0]["playlistVideoListRenderer"]["contents"].as_a ||
+ initial_data["response"]?.try &.["continuationContents"]["playlistVideoListContinuation"]["contents"].as_a).try &.each do |item|
+ if i = item["playlistVideoRenderer"]?
+ video_id = i["navigationEndpoint"]["watchEndpoint"]["videoId"].as_s
+ plid = i["navigationEndpoint"]["watchEndpoint"]["playlistId"].as_s
+ index = i["navigationEndpoint"]["watchEndpoint"]["index"].as_i64
+
+ thumbnail = i["thumbnail"]["thumbnails"][0]["url"].as_s
+ title = i["title"].try { |t| t["simpleText"]? || t["runs"]?.try &.[0]["text"]? }.try &.as_s || ""
+ author = i["shortBylineText"]?.try &.["runs"][0]["text"].as_s || ""
+ ucid = i["shortBylineText"]?.try &.["runs"][0]["navigationEndpoint"]["browseEndpoint"]["browseId"].as_s || ""
+ length_seconds = i["lengthSeconds"]?.try &.as_s.to_i
+ live = false
+
+ if !length_seconds
+ live = true
+ length_seconds = 0
+ end
+
+ videos << PlaylistVideo.new({
+ title: title,
+ id: video_id,
+ author: author,
+ ucid: ucid,
+ length_seconds: length_seconds,
+ published: Time.utc,
+ plid: plid,
+ live_now: live,
+ index: index - 1,
+ })
+ end
+ end
+
+ return videos
+end
+
def template_playlist(playlist)
html = <<-END_HTML
<h3>
diff --git a/src/invidious/search.cr b/src/invidious/search.cr
index 92baed0b..85fd024a 100644
--- a/src/invidious/search.cr
+++ b/src/invidious/search.cr
@@ -1,4 +1,19 @@
struct SearchVideo
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property ucid : String
+ property published : Time
+ property views : Int64
+ property description_html : String
+ property length_seconds : Int32
+ property live_now : Bool
+ property paid : Bool
+ property premium : Bool
+ property premiere_timestamp : Time?
+
def to_xml(auto_generated, query_params, xml : XML::Builder)
query_params["v"] = self.id
@@ -99,32 +114,27 @@ struct SearchVideo
def is_upcoming
premiere_timestamp ? true : false
end
-
- db_mapping({
- title: String,
- id: String,
- author: String,
- ucid: String,
- published: Time,
- views: Int64,
- description_html: String,
- length_seconds: Int32,
- live_now: Bool,
- paid: Bool,
- premium: Bool,
- premiere_timestamp: Time?,
- })
end
struct SearchPlaylistVideo
- db_mapping({
- title: String,
- id: String,
- length_seconds: Int32,
- })
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property length_seconds : Int32
end
struct SearchPlaylist
+ include DB::Serializable
+
+ property title : String
+ property id : String
+ property author : String
+ property ucid : String
+ property video_count : Int32
+ property videos : Array(SearchPlaylistVideo)
+ property thumbnail : String?
+
def to_json(locale, json : JSON::Builder)
json.object do
json.field "type", "playlist"
@@ -164,19 +174,19 @@ struct SearchPlaylist
end
end
end
-
- db_mapping({
- title: String,
- id: String,
- author: String,
- ucid: String,
- video_count: Int32,
- videos: Array(SearchPlaylistVideo),
- thumbnail: String?,
- })
end
struct SearchChannel
+ include DB::Serializable
+
+ property author : String
+ property ucid : String
+ property author_thumbnail : String
+ property subscriber_count : Int32
+ property video_count : Int32
+ property description_html : String
+ property auto_generated : Bool
+
def to_json(locale, json : JSON::Builder)
json.object do
json.field "type", "channel"
@@ -216,16 +226,6 @@ struct SearchChannel
end
end
end
-
- db_mapping({
- author: String,
- ucid: String,
- author_thumbnail: String,
- subscriber_count: Int32,
- video_count: Int32,
- description_html: String,
- auto_generated: Bool,
- })
end
alias SearchItem = SearchVideo | SearchChannel | SearchPlaylist
diff --git a/src/invidious/users.cr b/src/invidious/users.cr
index f3cfafa3..46bf8865 100644
--- a/src/invidious/users.cr
+++ b/src/invidious/users.cr
@@ -4,6 +4,20 @@ require "crypto/bcrypt/password"
MATERIALIZED_VIEW_SQL = ->(email : String) { "SELECT cv.* FROM channel_videos cv WHERE EXISTS (SELECT subscriptions FROM users u WHERE cv.ucid = ANY (u.subscriptions) AND u.email = E'#{email.gsub({'\'' => "\\'", '\\' => "\\\\"})}') ORDER BY published DESC" }
struct User
+ include DB::Serializable
+
+ property updated : Time
+ property notifications : Array(String)
+ property subscriptions : Array(String)
+ property email : String
+
+ @[DB::Field(converter: User::PreferencesConverter)]
+ property preferences : Preferences
+ property password : String?
+ property token : String
+ property watched : Array(String)
+ property feed_needs_update : Bool?
+
module PreferencesConverter
def self.from_rs(rs)
begin
@@ -13,31 +27,78 @@ struct User
end
end
end
-
- db_mapping({
- updated: Time,
- notifications: Array(String),
- subscriptions: Array(String),
- email: String,
- preferences: {
- type: Preferences,
- converter: PreferencesConverter,
- },
- password: String?,
- token: String,
- watched: Array(String),
- feed_needs_update: Bool?,
- })
end
struct Preferences
- module ProcessString
+ include JSON::Serializable
+ include YAML::Serializable
+
+ property annotations : Bool = CONFIG.default_user_preferences.annotations
+ property annotations_subscribed : Bool = CONFIG.default_user_preferences.annotations_subscribed
+ property autoplay : Bool = CONFIG.default_user_preferences.autoplay
+
+ @[JSON::Field(converter: Preferences::StringToArray)]
+ @[YAML::Field(converter: Preferences::StringToArray)]
+ property captions : Array(String) = CONFIG.default_user_preferences.captions
+
+ @[JSON::Field(converter: Preferences::StringToArray)]
+ @[YAML::Field(converter: Preferences::StringToArray)]
+ property comments : Array(String) = CONFIG.default_user_preferences.comments
+ property continue : Bool = CONFIG.default_user_preferences.continue
+ property continue_autoplay : Bool = CONFIG.default_user_preferences.continue_autoplay
+
+ @[JSON::Field(converter: Preferences::BoolToString)]
+ @[YAML::Field(converter: Preferences::BoolToString)]
+ property dark_mode : String = CONFIG.default_user_preferences.dark_mode
+ property latest_only : Bool = CONFIG.default_user_preferences.latest_only
+ property listen : Bool = CONFIG.default_user_preferences.listen
+ property local : Bool = CONFIG.default_user_preferences.local
+
+ @[JSON::Field(converter: Preferences::ProcessString)]
+ property locale : String = CONFIG.default_user_preferences.locale
+
+ @[JSON::Field(converter: Preferences::ClampInt)]
+ property max_results : Int32 = CONFIG.default_user_preferences.max_results
+ property notifications_only : Bool = CONFIG.default_user_preferences.notifications_only
+
+ @[JSON::Field(converter: Preferences::ProcessString)]
+ property player_style : String = CONFIG.default_user_preferences.player_style
+
+ @[JSON::Field(converter: Preferences::ProcessString)]
+ property quality : String = CONFIG.default_user_preferences.quality
+ property default_home : String = CONFIG.default_user_preferences.default_home
+ property feed_menu : Array(String) = CONFIG.default_user_preferences.feed_menu
+ property related_videos : Bool = CONFIG.default_user_preferences.related_videos
+
+ @[JSON::Field(converter: Preferences::ProcessString)]
+ property sort : String = CONFIG.default_user_preferences.sort
+ property speed : Float32 = CONFIG.default_user_preferences.speed
+ property thin_mode : Bool = CONFIG.default_user_preferences.thin_mode
+ property unseen_only : Bool = CONFIG.default_user_preferences.unseen_only
+ property video_loop : Bool = CONFIG.default_user_preferences.video_loop
+ property volume : Int32 = CONFIG.default_user_preferences.volume
+
+ module BoolToString
def self.to_json(value : String, json : JSON::Builder)
json.string value
end
def self.from_json(value : JSON::PullParser) : String
- HTML.escape(value.read_string[0, 100])
+ begin
+ result = value.read_string
+
+ if result.empty?
+ CONFIG.default_user_preferences.dark_mode
+ else
+ result
+ end
+ rescue ex
+ if value.read_bool
+ "dark"
+ else
+ "light"
+ end
+ end
end
def self.to_yaml(value : String, yaml : YAML::Nodes::Builder)
@@ -45,7 +106,20 @@ struct Preferences
end
def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : String
- HTML.escape(node.value[0, 100])
+ unless node.is_a?(YAML::Nodes::Scalar)
+ node.raise "Expected scalar, not #{node.class}"
+ end
+
+ case node.value
+ when "true"
+ "dark"
+ when "false"
+ "light"
+ when ""
+ CONFIG.default_user_preferences.dark_mode
+ else
+ node.value
+ end
end
end
@@ -67,33 +141,130 @@ struct Preferences
end
end
- json_mapping({
- annotations: {type: Bool, default: CONFIG.default_user_preferences.annotations},
- annotations_subscribed: {type: Bool, default: CONFIG.default_user_preferences.annotations_subscribed},
- autoplay: {type: Bool, default: CONFIG.default_user_preferences.autoplay},
- captions: {type: Array(String), default: CONFIG.default_user_preferences.captions, converter: ConfigPreferences::StringToArray},
- comments: {type: Array(String), default: CONFIG.default_user_preferences.comments, converter: ConfigPreferences::StringToArray},
- continue: {type: Bool, default: CONFIG.default_user_preferences.continue},
- continue_autoplay: {type: Bool, default: CONFIG.default_user_preferences.continue_autoplay},
- dark_mode: {type: String, default: CONFIG.default_user_preferences.dark_mode, converter: ConfigPreferences::BoolToString},
- latest_only: {type: Bool, default: CONFIG.default_user_preferences.latest_only},
- listen: {type: Bool, default: CONFIG.default_user_preferences.listen},
- local: {type: Bool, default: CONFIG.default_user_preferences.local},
- locale: {type: String, default: CONFIG.default_user_preferences.locale, converter: ProcessString},
- max_results: {type: Int32, default: CONFIG.default_user_preferences.max_results, converter: ClampInt},
- notifications_only: {type: Bool, default: CONFIG.default_user_preferences.notifications_only},
- player_style: {type: String, default: CONFIG.default_user_preferences.player_style, converter: ProcessString},
- quality: {type: String, default: CONFIG.default_user_preferences.quality, converter: ProcessString},
- default_home: {type: String, default: CONFIG.default_user_preferences.default_home},
- feed_menu: {type: Array(String), default: CONFIG.default_user_preferences.feed_menu},
- related_videos: {type: Bool, default: CONFIG.default_user_preferences.related_videos},
- sort: {type: String, default: CONFIG.default_user_preferences.sort, converter: ProcessString},
- speed: {type: Float32, default: CONFIG.default_user_preferences.speed},
- thin_mode: {type: Bool, default: CONFIG.default_user_preferences.thin_mode},
- unseen_only: {type: Bool, default: CONFIG.default_user_preferences.unseen_only},
- video_loop: {type: Bool, default: CONFIG.default_user_preferences.video_loop},
- volume: {type: Int32, default: CONFIG.default_user_preferences.volume},
- })
+ module FamilyConverter
+ def self.to_yaml(value : Socket::Family, yaml : YAML::Nodes::Builder)
+ case value
+ when Socket::Family::UNSPEC
+ yaml.scalar nil
+ when Socket::Family::INET
+ yaml.scalar "ipv4"
+ when Socket::Family::INET6
+ yaml.scalar "ipv6"
+ when Socket::Family::UNIX
+ raise "Invalid socket family #{value}"
+ end
+ end
+
+ def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : Socket::Family
+ if node.is_a?(YAML::Nodes::Scalar)
+ case node.value.downcase
+ when "ipv4"
+ Socket::Family::INET
+ when "ipv6"
+ Socket::Family::INET6
+ else
+ Socket::Family::UNSPEC
+ end
+ else
+ node.raise "Expected scalar, not #{node.class}"
+ end
+ end
+ end
+
+ module ProcessString
+ def self.to_json(value : String, json : JSON::Builder)
+ json.string value
+ end
+
+ def self.from_json(value : JSON::PullParser) : String
+ HTML.escape(value.read_string[0, 100])
+ end
+
+ def self.to_yaml(value : String, yaml : YAML::Nodes::Builder)
+ yaml.scalar value
+ end
+
+ def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : String
+ HTML.escape(node.value[0, 100])
+ end
+ end
+
+ module StringToArray
+ def self.to_json(value : Array(String), json : JSON::Builder)
+ json.array do
+ value.each do |element|
+ json.string element
+ end
+ end
+ end
+
+ def self.from_json(value : JSON::PullParser) : Array(String)
+ begin
+ result = [] of String
+ value.read_array do
+ result << HTML.escape(value.read_string[0, 100])
+ end
+ rescue ex
+ result = [HTML.escape(value.read_string[0, 100]), ""]
+ end
+
+ result
+ end
+
+ def self.to_yaml(value : Array(String), yaml : YAML::Nodes::Builder)
+ yaml.sequence do
+ value.each do |element|
+ yaml.scalar element
+ end
+ end
+ end
+
+ def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : Array(String)
+ begin
+ unless node.is_a?(YAML::Nodes::Sequence)
+ node.raise "Expected sequence, not #{node.class}"
+ end
+
+ result = [] of String
+ node.nodes.each do |item|
+ unless item.is_a?(YAML::Nodes::Scalar)
+ node.raise "Expected scalar, not #{item.class}"
+ end
+
+ result << HTML.escape(item.value[0, 100])
+ end
+ rescue ex
+ if node.is_a?(YAML::Nodes::Scalar)
+ result = [HTML.escape(node.value[0, 100]), ""]
+ else
+ result = ["", ""]
+ end
+ end
+
+ result
+ end
+ end
+
+ module StringToCookies
+ def self.to_yaml(value : HTTP::Cookies, yaml : YAML::Nodes::Builder)
+ (value.map { |c| "#{c.name}=#{c.value}" }).join("; ").to_yaml(yaml)
+ end
+
+ def self.from_yaml(ctx : YAML::ParseContext, node : YAML::Nodes::Node) : HTTP::Cookies
+ unless node.is_a?(YAML::Nodes::Scalar)
+ node.raise "Expected scalar, not #{node.class}"
+ end
+
+ cookies = HTTP::Cookies.new
+ node.value.split(";").each do |cookie|
+ next if cookie.strip.empty?
+ name, value = cookie.split("=", 2)
+ cookies << HTTP::Cookie.new(name.strip, value.strip)
+ end
+
+ cookies
+ end
+ end
end
def get_user(sid, headers, db, refresh = true)
@@ -103,8 +274,7 @@ def get_user(sid, headers, db, refresh = true)
if refresh && Time.utc - user.updated > 1.minute
user, sid = fetch_user(sid, headers, db)
user_array = user.to_a
-
- user_array[4] = user_array[4].to_json
+ user_array[4] = user_array[4].to_json # User preferences
args = arg_array(user_array)
db.exec("INSERT INTO users VALUES (#{args}) \
@@ -122,8 +292,7 @@ def get_user(sid, headers, db, refresh = true)
else
user, sid = fetch_user(sid, headers, db)
user_array = user.to_a
-
- user_array[4] = user_array[4].to_json
+ user_array[4] = user_array[4].to_json # User preferences
args = arg_array(user.to_a)
db.exec("INSERT INTO users VALUES (#{args}) \
@@ -166,7 +335,17 @@ def fetch_user(sid, headers, db)
token = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
- user = User.new(Time.utc, [] of String, channels, email, CONFIG.default_user_preferences, nil, token, [] of String, true)
+ user = User.new({
+ updated: Time.utc,
+ notifications: [] of String,
+ subscriptions: channels,
+ email: email,
+ preferences: Preferences.new(CONFIG.default_user_preferences.to_tuple),
+ password: nil,
+ token: token,
+ watched: [] of String,
+ feed_needs_update: true,
+ })
return user, sid
end
@@ -174,7 +353,17 @@ def create_user(sid, email, password)
password = Crypto::Bcrypt::Password.create(password, cost: 10)
token = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
- user = User.new(Time.utc, [] of String, [] of String, email, CONFIG.default_user_preferences, password.to_s, token, [] of String, true)
+ user = User.new({
+ updated: Time.utc,
+ notifications: [] of String,
+ subscriptions: [] of String,
+ email: email,
+ preferences: Preferences.new(CONFIG.default_user_preferences.to_tuple),
+ password: password.to_s,
+ token: token,
+ watched: [] of String,
+ feed_needs_update: true,
+ })
return user, sid
end
@@ -281,48 +470,6 @@ def subscribe_ajax(channel_id, action, env_headers)
end
end
-# TODO: Playlist stub, sync with YouTube for Google accounts
-# def playlist_ajax(video_ids, source_playlist_id, name, privacy, action, env_headers)
-# headers = HTTP::Headers.new
-# headers["Cookie"] = env_headers["Cookie"]
-#
-# html = YT_POOL.client &.get("/view_all_playlists", headers)
-#
-# cookies = HTTP::Cookies.from_headers(headers)
-# html.cookies.each do |cookie|
-# if {"VISITOR_INFO1_LIVE", "YSC", "SIDCC"}.includes? cookie.name
-# if cookies[cookie.name]?
-# cookies[cookie.name] = cookie
-# else
-# cookies << cookie
-# end
-# end
-# end
-# headers = cookies.add_request_headers(headers)
-#
-# if match = html.body.match(/'XSRF_TOKEN': "(?<session_token>[^"]+)"/)
-# session_token = match["session_token"]
-#
-# headers["content-type"] = "application/x-www-form-urlencoded"
-#
-# post_req = {
-# video_ids: [] of String,
-# source_playlist_id: "",
-# n: name,
-# p: privacy,
-# session_token: session_token,
-# }
-# post_url = "/playlist_ajax?#{action}=1"
-#
-# response = client.post(post_url, headers, form: post_req)
-# if response.status_code == 200
-# return JSON.parse(response.body)["result"]["playlistId"].as_s
-# else
-# return nil
-# end
-# end
-# end
-
def get_subscription_feed(db, user, max_results = 40, page = 1)
limit = max_results.clamp(0, MAX_ITEMS_PER_PAGE)
offset = (page - 1) * limit
diff --git a/src/invidious/videos.cr b/src/invidious/videos.cr
index 277c81f4..e7751fb0 100644
--- a/src/invidious/videos.cr
+++ b/src/invidious/videos.cr
@@ -222,30 +222,50 @@ VIDEO_FORMATS = {
}
struct VideoPreferences
- json_mapping({
- annotations: Bool,
- autoplay: Bool,
- comments: Array(String),
- continue: Bool,
- continue_autoplay: Bool,
- controls: Bool,
- listen: Bool,
- local: Bool,
- preferred_captions: Array(String),
- player_style: String,
- quality: String,
- raw: Bool,
- region: String?,
- related_videos: Bool,
- speed: (Float32 | Float64),
- video_end: (Float64 | Int32),
- video_loop: Bool,
- video_start: (Float64 | Int32),
- volume: Int32,
- })
+ include JSON::Serializable
+
+ property annotations : Bool
+ property autoplay : Bool
+ property comments : Array(String)
+ property continue : Bool
+ property continue_autoplay : Bool
+ property controls : Bool
+ property listen : Bool
+ property local : Bool
+ property preferred_captions : Array(String)
+ property player_style : String
+ property quality : String
+ property raw : Bool
+ property region : String?
+ property related_videos : Bool
+ property speed : Float32 | Float64
+ property video_end : Float64 | Int32
+ property video_loop : Bool
+ property video_start : Float64 | Int32
+ property volume : Int32
end
struct Video
+ include DB::Serializable
+
+ property id : String
+
+ @[DB::Field(converter: Video::JSONConverter)]
+ property info : Hash(String, JSON::Any)
+ property updated : Time
+
+ @[DB::Field(ignore: true)]
+ property captions : Array(Caption)?
+
+ @[DB::Field(ignore: true)]
+ property adaptive_fmts : Array(Hash(String, JSON::Any))?
+
+ @[DB::Field(ignore: true)]
+ property fmt_stream : Array(Hash(String, JSON::Any))?
+
+ @[DB::Field(ignore: true)]
+ property description : String?
+
module JSONConverter
def self.from_rs(rs)
JSON.parse(rs.read(String)).as_h
@@ -552,6 +572,7 @@ struct Video
def fmt_stream
return @fmt_stream.as(Array(Hash(String, JSON::Any))) if @fmt_stream
+
fmt_stream = info["streamingData"]?.try &.["formats"]?.try &.as_a.map &.as_h || [] of Hash(String, JSON::Any)
fmt_stream.each do |fmt|
if s = (fmt["cipher"]? || fmt["signatureCipher"]?).try { |h| HTTP::Params.parse(h.as_s) }
@@ -583,6 +604,9 @@ struct Video
fmt["url"] = JSON::Any.new("#{fmt["url"]}&host=#{URI.parse(fmt["url"].as_s).host}")
fmt["url"] = JSON::Any.new("#{fmt["url"]}&region=#{self.info["region"]}") if self.info["region"]?
end
+ # See https://github.com/TeamNewPipe/NewPipe/issues/2415
+ # Some streams are segmented by URL `sq/` rather than index, for now we just filter them out
+ fmt_stream.reject! { |f| !f["indexRange"]? }
fmt_stream.sort_by! { |f| f["width"]?.try &.as_i || 0 }
@adaptive_fmts = fmt_stream
return @adaptive_fmts.as(Array(Hash(String, JSON::Any)))
@@ -706,7 +730,7 @@ struct Video
end
def short_description
- info["shortDescription"]?.try &.as_s || ""
+ info["shortDescription"]?.try &.as_s? || ""
end
def hls_manifest_url : String?
@@ -748,30 +772,20 @@ struct Video
def session_token : String?
info["sessionToken"]?.try &.as_s?
end
+end
- db_mapping({
- id: String,
- info: {type: Hash(String, JSON::Any), converter: Video::JSONConverter},
- updated: Time,
- })
+struct CaptionName
+ include JSON::Serializable
- @captions : Array(Caption)?
- @adaptive_fmts : Array(Hash(String, JSON::Any))?
- @fmt_stream : Array(Hash(String, JSON::Any))?
+ property simpleText : String
end
struct Caption
- json_mapping({
- name: CaptionName,
- baseUrl: String,
- languageCode: String,
- })
-end
+ include JSON::Serializable
-struct CaptionName
- json_mapping({
- simpleText: String,
- })
+ property name : CaptionName
+ property baseUrl : String
+ property languageCode : String
end
class VideoRedirect < Exception
@@ -987,7 +1001,12 @@ def fetch_video(id, region)
raise info["reason"]?.try &.as_s || "" if !info["videoDetails"]?
- video = Video.new(id, info, Time.utc)
+ video = Video.new({
+ id: id,
+ info: info,
+ updated: Time.utc,
+ })
+
return video
end
@@ -1094,27 +1113,27 @@ def process_video_params(query, preferences)
controls ||= 1
controls = controls >= 1
- params = VideoPreferences.new(
- annotations: annotations,
- autoplay: autoplay,
- comments: comments,
- continue: continue,
- continue_autoplay: continue_autoplay,
- controls: controls,
- listen: listen,
- local: local,
- player_style: player_style,
+ params = VideoPreferences.new({
+ annotations: annotations,
+ autoplay: autoplay,
+ comments: comments,
+ continue: continue,
+ continue_autoplay: continue_autoplay,
+ controls: controls,
+ listen: listen,
+ local: local,
+ player_style: player_style,
preferred_captions: preferred_captions,
- quality: quality,
- raw: raw,
- region: region,
- related_videos: related_videos,
- speed: speed,
- video_end: video_end,
- video_loop: video_loop,
- video_start: video_start,
- volume: volume,
- )
+ quality: quality,
+ raw: raw,
+ region: region,
+ related_videos: related_videos,
+ speed: speed,
+ video_end: video_end,
+ video_loop: video_loop,
+ video_start: video_start,
+ volume: volume,
+ })
return params
end
generated by cgit v1.2.3 (git 2.39.1) at 2025-12-17 15:03:50 +0000