[Bug] Downloading revisions not working

I made a similar post about this one (https://lemmy.world/post/11134086) and it happened again, and hopefully the Dev can notice. (ping please)

Here’s the error that Perchance throws when I try to download revisions from https://perchance.org/power-generator-manager-10k-milestone:

Uncaught URIError: URI malformed
at decodeURI (see image)

EDIT: After some later investigation, I’ve found out that this does NOT affect earlier revisions (see this comment: https://lemmy.world/comment/8770044)

@perchance

  • BluePower
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    8 months ago

    That’s interesting. I didn’t know the hub also has the same revisions problem. I’ve been tinkering with that code for hours and this is what I made:

    function checkForMalformedRevs(patch, loglabel, startFrom) {
      let text;
      let dateRegex = [/^.* ([0-9][0-9]:[0-9][0-9]:[0-9][0-9]) GMT+.*/g, "$1"];
      console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Started.');
      patch.forEach((a,i) => {
        if (i < startFrom) return;
        let pi = diffStuff._patchesToRevision(patch, i);
        let x = 0;
        let offset = 0;
        try {
          for (x = 0; x < pi.length; x += 100) {
            offset = (!!pi.substring(x, x + 100).match(/%.$/g) * 1) + (!!pi.substring(x, x + 100).match(/%$/g) * 2); // checks if there's any "chopped" URI characters at the end of the chunk to prevent `URI malformed` errors just because of that
            text = decodeURI(pi.substring(x, x + 100 + offset));
            text = '';
            x += offset;
            offset = 0;
          }
          console.log('[' + loglabel + ']', Date().replace(...dateRegex), 'Checked:', i);
          x = 0;
        } catch (err) {
          console.error('[' + loglabel + ']', Date().replace(...dateRegex), 'Error:', i, 'at chunk', x + '-' + (x + 100 + offset), 'of', pi.length, err);
        }
      })
      console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Complete.');
    }
    checkForMalformedRevs(diffStuff.modelTextPatches, "modelTextPatches");
    checkForMalformedRevs(diffStuff.outputTemplatePatches, "outputTemplatePatches");
    

    There we go! It’s a greatly enhanced version of the checker tool. To reduce all the wait, the checker first processes a chunk of data, then if that chunk of data has been checked successfully, then the next chunk will be checked. (But still, if the revision data has around tens of thousands of characters it’s going to take seconds to process)

    The process roughly looks like this:

    "generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0A\n\n"
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking
    

    The checker will take a part of the data and then decodeURI’s them. Then if it’s successful, another chunk is also checked:

    "generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0A\n\n"
                                                      ~~~~~~~~!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking
    

    And there, once it’s URI malformed time, the checker stops for checking further chunks. Additionally, the logger not only throws the error, but also locates the chunk location where the error is throwed and shows when the check happened in time. You can also set on which index the checker should start checking with startFrom for easier debugging.

    Additional note: While testing, I also found that other “suspicious” (invalid) characters (including %9F) and the single percentage % are also affecting this whole revisions problem.

    (Also I might create a whole “utility” generator out of this so more people can contribute to this 😊)

    • VioneT@lemmy.worldM
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      8 months ago

      On that note, I duplicated the hub and deleted the old one, it did resolve the revisions problem (though the view count reset).

      • BluePower
        link
        fedilink
        English
        arrow-up
        1
        ·
        8 months ago

        What? With a hub that had over a thousand views? That’s remarkable.

        Also @[email protected], if there’s a way to backup and erase all the revisions somehow 🤔

        • VioneT@lemmy.worldM
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          8 months ago
          function checkForMalformedRevs(patch, loglabel, startFrom) {
            let text;
            let errors = []
            let dateRegex = [/^.* ([0-9][0-9]:[0-9][0-9]:[0-9][0-9]) GMT+.*/g, "$1"];
            console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Started.');
            patch.forEach((a,i) => {
              if (i < startFrom) return;
              let pi = diffStuff._patchesToRevision(patch, i);
              let x = 0;
              let offset = 0;
              try {
                for (x = 0; x < pi.length; x += 100) {
                  offset = (!!pi.substring(x, x + 100).match(/%.$/g) * 1) + (!!pi.substring(x, x + 100).match(/%$/g) * 2); // checks if there's any "chopped" URI characters at the end of the chunk to prevent `URI malformed` errors just because of that
                  text = pi.substring(x, x + 100 + offset)
                  let decoded = decodeURI(text);
                  decoded = '';
                  text = '';
                  x += offset;
                  offset = 0;
                }
                console.log('[' + loglabel + ']', Date().replace(...dateRegex), 'Checked:', i);
                x = 0;
              } catch (err) {
                console.error('[' + loglabel + ']', Date().replace(...dateRegex), 'Error:', i, 'at chunk', x + '-' + (x + 100 + offset), 'of', pi.length, err, 'Chunk Text:', text);
                errors.push({date: new Date(diffStuff.patches[i].creationTime), log: loglabel, revisionId: i, chunkRange: `${x}-${x + 100 + offset}`, revisionLength: pi.length, text: text});
                
              }
            })
            console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Complete.');
            console.table(errors, ['date', 'text']);
          }
          checkForMalformedRevs(diffStuff.modelTextPatches, 'modelTextPatches')
          // checkForMalformedRevs(diffStuff.outputTemplatePatches, 'outputTemplatePatches')
          

          Here is a code that also outputs the Date of the Revision as well as the Text that was throwing the error neatly in a table. It seems ONLY the modelTextPatches are throwing the malformeds.

          • BluePower
            link
            fedilink
            English
            arrow-up
            1
            ·
            8 months ago

            Oooh nice! I’ll test that later, and I’ll improve it even more when needed.