Computing Infrastructure: Difference between revisions

From Claremont MakerSpace
 
(16 intermediate revisions by the same user not shown)
Line 14: Line 14:
====<code>cms-net-svcs</code>/<code>salt</code>====
====<code>cms-net-svcs</code>/<code>salt</code>====
A vm on [[#cms-virt]] that hosts miscellaneous services, including:
A vm on [[#cms-virt]] that hosts miscellaneous services, including:
*the script that updates the HID door controllers.


*the [https://saltstack.com/ Salt] master. Salt is a Python based configuration management/remote task execution platform that I am abusing to remotely configure, install software, apply updates, and generally maintain the software on the laptops and desktops of the space. This allows me to declaratively define the desired state of the computers, rather than having each managed by hand, in theory resulting in more consistent and replicable behavior between computers. Having these properties means that each individual computer is less "important," and can be swapped out quickly in the event of a failure or upgrade.
*the [https://saltstack.com/ Salt] master. Salt is a Python based configuration management/remote task execution platform that I am abusing to remotely configure, install software, apply updates, and generally maintain the software on the laptops and desktops of the space. This allows me to declaratively define the desired state of the computers, rather than having each managed by hand, in theory resulting in more consistent and replicable behavior between computers. Having these properties means that each individual computer is less "important," and can be swapped out quickly in the event of a failure or upgrade.


*a [https://gitea.io/en-us/ Gitea] server, which provides git repository hosting for members, as well as hosting the various plumbing code for the space.
*a [https://gitea.io/en-us/ Gitea] server, which provides git repository hosting for members, as well as hosting the various plumbing code for the space.
*a [https://code.launchpad.net/~mvo/+junk/squid-deb-proxy Squid-deb-proxy] server, which provides package caching for debian and raspbian. You should be able to use it by just installing the <code>squid-deb-proxy</code> package.
 
*a [https://wiki.debian.org/AptCacherNg Apt-Cacher-NG] server, which provides package caching for debian and raspbian. You should be able to use it by just installing the <code>squid-deb-proxy-client</code> package.


====<code>cms-www</code>====
====<code>cms-www</code>====
A vm on [[#cms-virt]] that runs nginx as a reverse proxy for various web services accessible from the outside world.
A vm on [[#cms-virt]] that runs [https://nginx.org/ nginx] as a reverse proxy for various web services accessible from the outside world.
 
* [https://manage.claremontmakerspace.org CMSManage] <sup>[https://git.claremontmakerspace.org/CMS/cmsmanage <nowiki>[source]</nowiki>]</sup> A custom Python/Django app that provides various member and administrative services, as well as syncing data between various external sources (see [[#Data]] for more details).
* https://git.claremontmakerspace.org/: Git repository hosting, including a lot of the internal code for the makerspace. (reverse proxy from cms-net-svcs)
* https://3dprint.claremontmakerspace.org/: Remote monitoring for the makerspace's 3D printers, providing webcams and status from OctoPrint. Source code lives here: https://git.claremontmakerspace.org/adam.goldsmith/PrinterStatus/


====<code>cms-asterisk</code>====
====<code>cms-asterisk</code>====
A vm on [[#cms-virt]] that runs asterisk for our phone and voicemail systems.
A vm on [[#cms-virt]] that runs [https://www.asterisk.org/ Asterisk] for our phone and voicemail systems.


====<code>cms-hass</code>====
====<code>cms-hass</code>====
A vm on [[#cms-virt]] that runs home assistant for monitoring a variety of building monitoring services.
A vm on [[#cms-virt]] that runs [https://www.home-assistant.io/ Home Assistant] for monitoring a variety of building monitoring services
 
==== <code>cms-analytics</code> ====
A vm in [[#cms-virt]]  that runs [https://www.influxdata.com/products/influxdb-overview/ InfluxDB] and [https://grafana.com/ Grafana] for storing and graphing data.


====<code>cms-fah</code>====
====<code>cms-fah</code>====
A vm on [[#cms-virt]] that is running folding@home. In March of 2020 we have dedicated most of the resources on [[#cms-virt]] to this VM to help find therapies and cures for COVID-19.
A vm on [[#cms-virt]] that is running [https://foldingathome.org/ Folding@home]. For several months starting in March of 2020 we dedicated most of the resources on [[#cms-virt]] to this VM to help find therapies and cures for COVID-19. Currently disabled at time of writing.


===<code>cms-games</code>===
===<code>cms-games</code>===
The other physical server we run, a Dell R210 II. This one hosts a few game servers, currently [https://www.minecraft.net/ Minecraft] and [https://www.factorio.com/ Factorio]. On the members network, these are available at <code>cms-games.sawtooth.claremontmakerspace.org</code>, and <code>games.claremontmakerspace.org</code> from the internet (put that address into your game client).
The other physical server we run, a Dell R210 II. This one hosts a few game servers, currently [https://www.minecraft.net/ Minecraft] and [https://www.factorio.com/ Factorio]. On the members network, these are available at <code>cms-games.sawtooth.claremontmakerspace.org</code>, and <code>games.claremontmakerspace.org</code> from the internet (put that address into your game client).


===<code>octopi-taz-6</code> and <code>octopi-lulzbot-mini</code>===
===<code>octopi-taz-6</code>, <code>octopi-lulzbot-mini</code>, and <code>octopi-anycubic-c</code>===
These are Raspberry Pis connected to the [[3D Printers]] in the [[Digital Fab. and Electronics Lab]], which run [https://octoprint.org/ OctoPrint], providing a web GUI for the 3D printers. While on the members network, you can connect to them at:
These are Raspberry Pis connected to the [[3D Printers]] in the [[Digital Fab. and Electronics Lab]], which run [https://octoprint.org/ OctoPrint], providing a web GUI for the 3D printers. While on the members network, you can connect to them at:


*http://octopi-taz-6.sawtooth.claremontmakerspace.org
*http://octopi-taz-6.sawtooth.claremontmakerspace.org
*http://octopi-lulzbot-mini.sawtooth.claremontmakerspace.org
*http://octopi-lulzbot-mini.sawtooth.claremontmakerspace.org
*http://octopi-anycubic-c.sawtooth.claremontmakerspace.org


===<code>hosting.tardisventures.net</code>===
===<code>hosting.tardisventures.net</code>===
Runs most of the off-site accessible/website stuff, including the WordPress instance on our main site https://claremontmakerspace.org, the wiki (here), the inventory site at https://inventory.claremontmakerspace.org, and the ticket system at https://tickets.claremontmakerspace.org.
Runs most of the off-site accessible/website stuff.
* https://claremontmakerspace.org: WordPress instance on our main site  
* https://wiki.claremontmakerspace.org: the wiki (here)
* https://inventory.claremontmakerspace.org: inventory/asset tracking
* https://tickets.claremontmakerspace.org: the ticket system
* https://discourse.claremontmakerspace.org: forums
* https://inventree.claremontmakerspace.org: electronic components collection database
* https://mailman.claremontmakerspace.org/: Mailing list management
 
The wiki, inventory, ticketing, Discourse, and InvenTree are all running in [https://www.docker.com/ Docker] containers for ease of management, as it allows for declarative configuration, easier updating, and simplified dependency management compared to running directly on <code>hosting</code>.
 
<uml>
title CMS services
left to right direction
node "cms-virt" {
    node "cms-ucs" {
        agent "Univention Corporate Server" as ucs
    }
 
    node "cms-net-svcs" {
        agent salt
        agent MariaDB
        agent gitea
    }
 
    node "cms-www" {
        agent printerStatus
        agent CMSManage
        agent nginx
    }
 
    node "cms-asterisk"
    node "cms-hass"
}
 
node "cms-games" {
    agent Minecraft
    agent Factorio
}
 
frame "3D Printers" as 3dprinters {
    node "octopi-taz-6"
    node "octopi-lulzbot-mini"
    node "octopi-anycubic-c"
}
 
cloud internet {
    node hosting.tardisventures.net as hosting {
        agent Wordpress
        agent MediaWiki
        agent osTicket
        agent "Snipe-IT"
        agent Discourse
        agent InvenTree
    }
}
 
3dprinters --> printerStatus
printerStatus -- nginx
 
CMSManage -- MariaDB
 
gitea -- nginx
 
ucs -- nginx : "self-service"
 
nginx -- internet
 
Factorio -- internet
Minecraft -- internet
</uml>
 
=Data=
Member signups, event registration, billing, etc. happen in [https://membershipworks.com MembershipWorks]. A custom Python/Django application called [https://manage.claremontmakerspace.org CMSManage] <sup>[https://git.claremontmakerspace.org/CMS/cmsmanage <nowiki>[source]</nowiki>]</sup> handles synchronizing this data with on-site systems, including the Windows domain (on [[#cms-ucs|cms-ucs]]) and door access controllers.
 
<uml>
title Data Flows Through CMS Services
left to right direction
cloud internet {
    node membershipworks.com
    node hosting.tardisventures.net {
        agent Mailman3
        agent "mailman-sync"
    }
}
 
collections "HID door controllers" as doorControllers
node "cms-ucs" as cms_ucs
 
node "cms-net-svcs" {
    database CMS_Database
}
 
node "cms-www" {
    package CMSManage {
        agent doorUpdater
        agent hidEvents
        agent scrapeMembershipworks
        agent ucsAccounts
    }
}
 
membershipworks.com --> scrapeMembershipworks
scrapeMembershipworks --> CMS_Database
 
CMS_Database --> doorUpdater
CMS_Database --> ucsAccounts
 
doorUpdater --> doorControllers
ucsAccounts --> cms_ucs
 
doorControllers --> hidEvents
hidEvents --> CMS_Database
 
CMS_Database <--> CMSManage


The wiki, inventory, and ticketing are all running in [https://www.docker.com/ Docker] containers for ease of management, as it allows for declarative configuration, easier updating, and simplified dependency management compared to running directly on <code>hosting</code>.
CMSManage --> "mailman-sync"
"mailman-sync" --> Mailman3
</uml>


=Network=
=Network=
The Claremont MakerSpace has high speed Internet access (200x20Mb) from Comcast. Our internal network is based on  Ubiquiti Unifi equipment with four UniFi 48 port Power Over Ethernet (POE) switches connected together via a 10Gb multimode fiber optic ring. [[File:CMS Network Diagram.png|frameless|border]] WiFi is provided throughout the building and property by five WiFi access points located throughout the building.
<br />
{{Flag-TODO}}
{{Flag-TODO}}
[[File:CMS Network Diagram.png|thumb|Network Diagram]]
The Claremont MakerSpace has high speed Internet access (200x20Mb) from Comcast. Our internal network is based on  Ubiquiti Unifi equipment with four UniFi 48 port Power Over Ethernet (POE) switches connected together via a 10Gb multimode fiber optic ring. WiFi is provided throughout the building and property by five WiFi access points located in various places in the building.

Latest revision as of 04:28, 1 April 2024

Computers

There are several laptops and desktops at the space. At the time of writing, they are mostly running Windows 10, with a few exceptions still running 7 for compatibility or licensing reasons which will hopefully be resolved in the near future. They are all joined to the SAWTOOTH domain, for which the domain controller is #cms-ucs. They are somewhat centrally managed via a salt master running on #cms-net-svcs.

Currently, they are set up to authenticate against #cms-ucs, with roaming profiles and folder redirection in place so that files are synchronized across computers in the domain. You can also get access to your files from another computer, as long as you are connected to the CMS-Members network (or Ethernet at the space). On Windows, for example, this could be done by entering \\ucs.sawtooth.claremontmakerspace.org\YOUR_USERNAME_HERE in the address bar of the file explorer, then typing your username and password when prompted.

Servers

cms-virt

The primary on-site server we run; it is a Cisco UCSC-C220-M3S rack-mount server with 16 processors and 64GB of RAM, living in the electrical closet. It hosts a number of Virtual Machines (VMs), primarily relating to the internal workings of the space. It runs debian, with libvirt/KVM for virtualization.

cms-ucs

A vm on #cms-virt that hosts the Univention Corporate Server (UCS), which is the domain controller and file share server for all of the Windows machines, as well as the LDAP authentication provider for various services.

cms-net-svcs/salt

A vm on #cms-virt that hosts miscellaneous services, including:

  • the Salt master. Salt is a Python based configuration management/remote task execution platform that I am abusing to remotely configure, install software, apply updates, and generally maintain the software on the laptops and desktops of the space. This allows me to declaratively define the desired state of the computers, rather than having each managed by hand, in theory resulting in more consistent and replicable behavior between computers. Having these properties means that each individual computer is less "important," and can be swapped out quickly in the event of a failure or upgrade.
  • a Gitea server, which provides git repository hosting for members, as well as hosting the various plumbing code for the space.
  • a Apt-Cacher-NG server, which provides package caching for debian and raspbian. You should be able to use it by just installing the squid-deb-proxy-client package.

cms-www

A vm on #cms-virt that runs nginx as a reverse proxy for various web services accessible from the outside world.

cms-asterisk

A vm on #cms-virt that runs Asterisk for our phone and voicemail systems.

cms-hass

A vm on #cms-virt that runs Home Assistant for monitoring a variety of building monitoring services

cms-analytics

A vm in #cms-virt that runs InfluxDB and Grafana for storing and graphing data.

cms-fah

A vm on #cms-virt that is running Folding@home. For several months starting in March of 2020 we dedicated most of the resources on #cms-virt to this VM to help find therapies and cures for COVID-19. Currently disabled at time of writing.

cms-games

The other physical server we run, a Dell R210 II. This one hosts a few game servers, currently Minecraft and Factorio. On the members network, these are available at cms-games.sawtooth.claremontmakerspace.org, and games.claremontmakerspace.org from the internet (put that address into your game client).

octopi-taz-6, octopi-lulzbot-mini, and octopi-anycubic-c

These are Raspberry Pis connected to the 3D Printers in the Digital Fab. and Electronics Lab, which run OctoPrint, providing a web GUI for the 3D printers. While on the members network, you can connect to them at:

hosting.tardisventures.net

Runs most of the off-site accessible/website stuff.

The wiki, inventory, ticketing, Discourse, and InvenTree are all running in Docker containers for ease of management, as it allows for declarative configuration, easier updating, and simplified dependency management compared to running directly on hosting.

Data

Member signups, event registration, billing, etc. happen in MembershipWorks. A custom Python/Django application called CMSManage [source] handles synchronizing this data with on-site systems, including the Windows domain (on cms-ucs) and door access controllers.

Network

This section has been marked as TODO.

Network Diagram

The Claremont MakerSpace has high speed Internet access (200x20Mb) from Comcast. Our internal network is based on Ubiquiti Unifi equipment with four UniFi 48 port Power Over Ethernet (POE) switches connected together via a 10Gb multimode fiber optic ring. WiFi is provided throughout the building and property by five WiFi access points located in various places in the building.