This weekend, TASBot will finally take its talents into the modern gaming era, showing off expert-level Super Mario Maker 2 gameplay on an actual Switch during the livestreamed Awesome Games Done Quick speedrunning marathon. And this time, the TASBot team is taking pains to make sure no one else can copy its method—to hopefully avoid Nintendo's potential legal ire in the process.
Flipping the Switch
The effort to let a Linux computer take external control of a Switch game began a bit inadvertently back in 2018, when the TASBot team attempted to partner with the AbleGamers charity. Their goal was to create an Arduino interface that would allow inputs (and pre-recorded input macros) from any controller to be re-mapped into input signals for any console interface.
While that AbleGamers effort eventually fizzled out, it did lead to a generalized Linux-to-Switch controller interface that was published on GitHub. At the same time, other efforts like CommunityController's "Twitch plays Nintendo Switch" were using similar concepts to let a Twitch chat room take control of live Switch gameplay (a la 2014's "Twitch Plays Pokemon" phenomenon).
While these kinds of efforts were fun for random tinkering, they utterly lacked the frame-perfect precision necessary for a successful replay of a pre-recorded, tool-assisted speedrun. "We saw massive inconsistencies," TASBot maintainer Allan "dwangoAC" Cecil told Ars about TASBot testing on the Switch in 2018. "Replay device precision was impossible... TASBot is a player piano—he's playing back a predefined sequence of button presses—but if he doesn't know when to send those button presses, it'll never work."
-
A circa 2016 version of TASBot shows off his new NES control display
-
P4plus2 carries TASBot down the AGDQ hallway to his big show in 2016.
-
TASBot circa 2019 sits at the ready in the GDQ practice room
-
A visualization of the setup that let TASBot pump arbitrary video through an SNES and streaming audio through an NES.Brian Mulligan
-
TASBot sits next to MASHBot, which can actually tap a stylus on a Nintendo DS screen.
-
Last minute adjustments before TASBot's 2019 AGDQ performance.
By 2019, multiple TASBot team members were working in parallel to try to solve this seemingly intractable Switch timing problem. One branch of effort even tried to insert a "shim layer" onto a hacked Switch console to force the external input timing to line up with the in-game timing, but "we didn't get far because it's against our ethos to modify the console," Cecil said.
At the same time, TASBot team member KNfLrPn was "using the semi-working system to help test [Super Mario Maker 2] tech for other [efforts]," they recently told Ars. "So while doing that I kept trying different things just in case, and eventually found a combination of multiple pieces that worked together [to fix the timing problem]."
Prior to that first successful test in December, there was "about five months on-and-off of trying different approaches, different code, different hardware," KNfLrPn added. "Until it worked, we had no idea if it was possible (and actually suspected that it wasn't)."
Approaching the starting line
Though TASBot has taken the first step to breaking open robotic Switch play, its method still isn't perfect. For one, Cecil says the hardware still isn't precise enough for games that require analog input.
In testing on Breath of the Wild, for instance, the team tried recording a simple input macro of Link jumping off a tower. But Cecil said slight, frame-level differences between the Linux recording and the controller polling rate during playback led to butterfly effect-style chaos, such that "loading the same savestate and playing [the input] back would result in us landing in a different spot, sometimes substantially so." Using digital inputs on a more deterministic game like Super Mario Maker 2 eliminates those problems, Cecil added.
Playing on the Switch also means the TASBot team doesn't have the benefit of recording its inputs on robust, TAS-configured emulators, which allow for easy pausing, editing, and re-recording of frame-perfect input sequences that can create literally superhuman performance. On the Switch, thus far "there aren't any tools to make this fast," Cecil said. "This was done laboriously by hand and isn't easy to replicate."
For this weekend's AGDQ demonstration, KNfLrPn specifically designed a level to take these limitations into account; for each level section, they "include[d] a spot where I could get a consistent starting point (You might see in each part there's some kind of wall I could press against)."
From those safe spots, KNfLrPn said they could "start with a guess on which buttons to press for how long, try it, see what happens, adjust, and iterate over and over" until they reached the next safe spot. By playing a string of successfully recorded sections back from the start, KNfLrPn could then get back to any safe spot to continue the trial-and-error process.
Without the use of emulator tools, recording a few successful minutes of Switch gameplay took "several hours of trial and error, resetting each section and trying something slightly different each time," KNfLrPn said. "It was also 'only' several hours because I specifically designed each section to be easy to reset. Doing it with a 'real' level would be even more tedious."
Secrets and lawyers
Compared to some of our previous explainers on TASBot, you may have noticed I didn't go into detail on the actual method the TASBot team used to solve its Switch timing problem. That's because the solution—which requires a bit of extra video signal analysis hardware that the team is keeping hidden in a literal "black box"—could lead copycats to unleash utter chaos on some active Switch online leaderboards, including the recently launched Ninji Speedrun competitions on Super Mario Maker 2.
"This has a higher risk of widespread damage because Nintendo has not always been attentive to illegitimate leaderboard entries," Cecil said. "If a troll wanted to, they could make it impossible for a human to obtain the fastest time in the regularly released Ninji speedrun levels."
A TASBot team member (who asked to remain anonymous) went even further. "The knowledge of how to do this can and will affect records on some of the most difficult levels in the game... This tool could allow an individual the ability to trial and error their way through a level, and then release a perfect run to anyone on the Internet that also wishes to 'beat' a level. This would ruin the experience players have, as no one would know if a top record on a level is real or if it was done by a user in a malicious way."
The team's concern for methodological secrecy also mean this is one of the first TASBot projects where the team won't be releasing its source code publicly. That's a decision Cecil says he didn't take lightly. "As the President of the North Bay Linux Users' Group and an advocate for open source software, I always ensure we release what we create as open source and open hardware so others can replicate it," Cecil said. "In this case, doing so is both risky and unwise due to the potential damage to the community... I made this decision after consulting with a diverse range of community members and experts, including paying for a consultation with a lawyer who specializes in video game lawsuits."
-
A 3D model showing TASBot's new prototype design.
-
Potential color scheme for the new TASBot
-
A look inside TASBot's innards
-
TASBot's upcoming LED eyes can show multiple colors and emotions.
-
An alternate color design for the Yettee TASBot shirt.
-
New logo design.
(The threat of legal complications has also led the TASBot team to redesign the robot's public-facing shell, which is built off a repurposed NES R.O.B. controller. A new prototype design retains the same general feeling while being distinct enough for independent trademarking by the charity-focused TASBot L3C, Cecil said. The new design is also featured on an exclusive Yetee t-shirt, with proceeds going to the Prevent Cancer Foundation).
Who to tell
Cecil, who works as a security consultant at Bishop Fox, said the team discussed reaching out to Nintendo before publicizing its Switch-control method, but it "chose to not poke the bear." That's in part because controlling the Switch with a robot—using completely unmodified Switch hardware and software and standard controller input signals through the USB port—doesn't completely match the usual definition of a "security vulnerability."
"Nintendo has a vulnerability disclosure program, but the methods we're using don't fall under the category of issues that can be reported," Cecil said. "We're using Nintendo's hardware in a fully standards-complaint way and there is no way for them to prevent what we are doing without disabling all external devices. In other words, most companies don't have a big enough imagination to contemplate something so out-of-the-box, and there is no way for them to do anything about it even if we did provide a disclosure. So we have to take other precautions."
Cecil said he and the TASBot team have gone back and forth over whether to even show TASBot controlling the Switch at AGDQ (or last week's similar MAGFast speedrun marathon). Now, though, Cecil says he thinks there are enough precautions in place to "keep low effort script kiddies and copy and paste trolls from ruining the fun for everyone else." AGDQ management initially pulled the Switch demo due to time constraints, Cecil said, but the event recently added it back in as a donation incentive following the standard Super Mario Maker 2 demonstration on Saturday night.
Not everyone is convinced the TASBot team's efforts at operational secrecy will be enough, though. "Lots of 'fake' input devices have been made that have the possibility for [Switch] TASing, but none (that we've seen) have used [that potential]," TASBot team member Britmob said. "Frankly, I'm surprised no one else came up with it. But I don't expect people to be very far behind us, especially if they see if it's possible, regardless of us not disclosing methods.""We will not be discussing the abuse concerns we have during the presentation because we do not want to draw attention to them," Cecil said. "But we do want to be open about the risks we face. Pretending the risks don't exist will not help us.
"We want to continue to show what we consider art at charity events and it's important to us to find the right balance of openness even on content designed for newer consoles," he added. "The TASBot community has become so much larger than anything I could have hoped for or done on my own and I truly feel like I've been given a wonderful opportunity to live beyond myself."
Bagikan Berita Ini
0 Response to "Inside TASBot’s semi-secret, probably legal effort to control the Nintendo Switch - Ars Technica"
Post a Comment