Skip to content
Snippets Groups Projects
  1. Apr 10, 2023
    • Joey Hess's avatar
      filter out control characters in warning messages · 3290a09a
      Joey Hess authored
      Converted warning and similar to use StringContainingQuotedPath. Most
      warnings are static strings, some do refer to filepaths that need to be
      quoted, and others don't need quoting.
      
      Note that, since quote filters out control characters of even
      UnquotedString, this makes all warnings safe, even when an attacker
      sneaks in a control character in some other way.
      
      When json is being output, no quoting is done, since json gets its own
      quoting.
      
      This does, as a side effect, make warning messages in json output not
      be indented. The indentation is only needed to offset warning messages
      underneath the display of the file they apply to, so that's ok.
      
      Sponsored-by: Brett Eisenberg on Patreon
      3290a09a
  2. Mar 27, 2023
    • Joey Hess's avatar
      addurl, importfeed: Fix failure when annex.securehashesonly is set · 24ae4b29
      Joey Hess authored
      The temporary URL key used for the download, before the real key is
      generated, was blocked by annex.securehashesonly.
      
      Fixed by passing the Backend that will be used for the final key into
      runTransfer. When a Backend is provided, have preCheckSecureHashes
      check that, rather than the key being transferred.
      
      Sponsored-by: unqueued on Patreon
      24ae4b29
  3. Mar 17, 2023
  4. Jun 28, 2022
  5. Oct 06, 2021
    • Joey Hess's avatar
      remove unused imports · 65cf6fb7
      Joey Hess authored
      65cf6fb7
    • Joey Hess's avatar
      remove old and uncessary sanity check · f2846d80
      Joey Hess authored
      All backends use genKeyName, which uses preSanitizeKeyName, which will
      escape newlines.
      
      I suspect this newline prevention was added back when git-annex was
      reading the output of sha1sum..
      
      Sponsored-by: Kevin Mueller on Patreon
      f2846d80
    • Joey Hess's avatar
      convert Key to ShortByteString · 19e78816
      Joey Hess authored
      This adds the overhead of a copy when serializing and deserializing keys.
      I have not benchmarked much, but runtimes seem barely changed at all by that.
      
      When a lot of keys are in memory, it improves memory use.
      
      And, it prevents keys sometimes getting PINNED in memory and failing to GC,
      which is a problem ByteString has sometimes. In particular, git-annex sync
      from a borg special remote had that problem and this improved its memory
      use by a large amount.
      
      Sponsored-by: Shae Erisson on Patreon
      19e78816
  6. Jul 27, 2021
    • Joey Hess's avatar
      move verifyKeyContent to Annex.Verify · de482c7e
      Joey Hess authored
      The goal is that Database.Keys be able to use it; it can't use
      Annex.Content.Presence due to an import loop.
      
      Several other things also needed to be moved to Annex.Verify as a
      conseqence.
      de482c7e
  7. Feb 09, 2021
    • Joey Hess's avatar
      incremental checksum on download from ssh or p2p · 62e152f2
      Joey Hess authored
      Checksum as content is received from a remote git-annex repository, rather
      than doing it in a second pass.
      
      Not tested at all yet, but I imagine it will work!
      
      Not implemented for any special remotes, and also not implemented for
      copies from local remotes. It may be that, for local remotes, it will
      suffice to use rsync, rely on its checksumming, and simply return Verified.
      (It would still make a checksumming pass when cp is used for COW, I guess.)
      62e152f2
  8. Oct 29, 2020
  9. Jul 29, 2020
    • Joey Hess's avatar
      external backends wip · f75be321
      Joey Hess authored
      It's able to start them up, the only thing not implemented is generating
      and verifying keys. And, the key translation for HasExt.
      f75be321
  10. Jul 20, 2020
    • Joey Hess's avatar
      rename getKey to genKey · c4cc2cdf
      Joey Hess authored
      for consistency with external backend protocol
      c4cc2cdf
    • Joey Hess's avatar
      move cryptographicallySecure into Backend type · 17274372
      Joey Hess authored
      This is groundwork for external backends, but also makes sense to keep
      this information with the rest of a Backend's implementation.
      
      Also, removed isVerifiable. I noticed that the same information is
      encoded by whether a Backend implements verifyKeyContent or not.
      17274372
  11. May 15, 2020
    • Joey Hess's avatar
      change retrieveExport and getKey to throw exception · 3334d383
      Joey Hess authored
      retrieveExport is part of ongoing transition to make remote methods
      throw exceptions, rather than silently hide them.
      
      getKey very rarely fails, and when it does it's always for the same reason
      (user configured annex.backend to url for some reason). So, this will
      avoid dealing with Nothing everywhere it's used.
      
      This commit was sponsored by Ilya Shlyakhter on Patreon.
      3334d383
  12. Nov 22, 2019
    • Joey Hess's avatar
      cache the serialization of a Key · 81d40221
      Joey Hess authored
      This will speed up the common case where a Key is deserialized from
      disk, but is then serialized to build eg, the path to the annex object.
      
      Previously attempted in 4536c93b
      and reverted in 96aba8ef.
      The problems mentioned in the latter commit are addressed now:
      
      Read/Show of KeyData is backwards-compatible with Read/Show of Key from before
      this change, so Types.Distribution will keep working.
      
      The Eq instance is fixed.
      
      Also, Key has smart constructors, avoiding needing to remember to update
      the cached serialization.
      
      Used git-annex benchmark:
        find is 7% faster
        whereis is 3% faster
        get when all files are already present is 5% faster
      Generally, the benchmarks are running 0.1 seconds faster per 2000 files,
      on a ram disk in my laptop.
      81d40221
  13. Jun 25, 2019
    • Joey Hess's avatar
      plumb MeterUpdate into getKey · 8355dba5
      Joey Hess authored
      No behavior changes, but this shows everywhere that a progress meter
      could be displayed when hashing a file to add to the annex.
      
      Many of the places don't make sense to display a progress meter though,
      eg when importing the copy of the file probably swamps the hashing of
      the file.
      8355dba5
  14. Mar 13, 2019
    • Joey Hess's avatar
      update licenses from GPL to AGPL · 40ecf58d
      Joey Hess authored
      This does not change the overall license of the git-annex program, which
      was already AGPL due to a number of sources files being AGPL already.
      
      Legally speaking, I'm adding a new license under which these files are
      now available; I already released their current contents under the GPL
      license. Now they're dual licensed GPL and AGPL. However, I intend
      for all my future changes to these files to only be released under the
      AGPL license, and I won't be tracking the dual licensing status, so I'm
      simply changing the license statement to say it's AGPL.
      
      (In some cases, others wrote parts of the code of a file and released it
      under the GPL; but in all cases I have contributed a significant portion
      of the code in each file and it's that code that is getting the AGPL
      license; the GPL license of other contributors allows combining with
      AGPL code.)
      40ecf58d
  15. Jan 16, 2019
    • Joey Hess's avatar
      Revert "cache the serialization of a Key" · 96aba8ef
      Joey Hess authored
      This reverts commit 4536c93b.
      
      That broke Read/Show of a Key, and unfortunately Key is read in at least
      one place; the GitAnnexDistribution data type.
      
      It would be worth bringing this optimisation back, but it would need
      either a custom Read/Show instance that preserves back-compat, or
      wrapping Key in a data type that contains the serialization, or changing
      how GitAnnexDistribution is serialized.
      
      Also, the Eq instance would need to compare keys with and without a
      cached seralization the same.
      96aba8ef
  16. Jan 14, 2019
    • Joey Hess's avatar
      cache the serialization of a Key · 4536c93b
      Joey Hess authored
      This will speed up the common case where a Key is deserialized from
      disk, but is then serialized to build eg, the path to the annex object.
      
      It means that every place a Key has any of its fields changed, the cache
      has to be dropped. I've grepped and found them all. But, it would be
      better to avoid that gotcha somehow..
      4536c93b
  17. Jan 11, 2019
  18. Dec 05, 2017
  19. May 09, 2017
  20. Mar 01, 2017
  21. Feb 24, 2017
    • Joey Hess's avatar
      add KeyVariety type · 9c465035
      Joey Hess authored
      Where before the "name" of a key and a backend was a string, this makes
      it a concrete data type.
      
      This is groundwork for allowing some varieties of keys to be disabled
      in file2key, so git-annex won't use them at all.
      
      Benchmarks ran in my big repo:
      
      old git-annex info:
      
      real	0m3.338s
      user	0m3.124s
      sys	0m0.244s
      
      new git-annex info:
      
      real	0m3.216s
      user	0m3.024s
      sys	0m0.220s
      
      new git-annex find:
      
      real	0m7.138s
      user	0m6.924s
      sys	0m0.252s
      
      old git-annex find:
      
      real	0m7.433s
      user	0m7.240s
      sys	0m0.232s
      
      Surprising result; I'd have expected it to be slower since it now parses
      all the key varieties. But, the parser is very simple and perhaps
      sharing KeyVarieties uses less memory or something like that.
      
      This commit was supported by the NSF-funded DataLad project.
      9c465035
  22. Jan 20, 2016
  23. Dec 15, 2015
  24. Dec 09, 2015
  25. Dec 07, 2015
    • Joey Hess's avatar
      support pointer files · 664cc987
      Joey Hess authored
      Backend.lookupFile is changed to always fall back to catKey when
      operating on a file that's not a symlink.
      
      catKey is changed to understand pointer files, as well as annex symlinks.
      
      Before, catKey needed a file mode witness, to be sure it was looking at a
      symlink. That was complicated stuff. Now, it doesn't actually care if a
      file in git is a symlink or not; in either case asking git for the content
      of the file will get the pointer to the key.
      
      This does mean that git-annex will treat a link
      foo -> WORM--bar as a git-annex file, and also treats
      a regular file containing annex/objects/WORM--bar as a git-annex file.
      
      Calling catKey could make git-annex commands need to do more work than
      before. This would especially be the case if a repo contained many regular
      files, and only a few annexed files, as now git-annex will need to ask
      git about the contents of the regular files.
      664cc987
  26. Jan 21, 2015
  27. Jul 30, 2014
    • Joey Hess's avatar
      only chunk stable keys · 89416ba2
      Joey Hess authored
      The content of unstable keys can potentially be different in different
      repos, so eg, resuming a chunked upload started by another repo would
      corrupt data.
      89416ba2
  28. Jul 27, 2014
  29. Jul 17, 2014
    • Joey Hess's avatar
      · e213ef31
      Joey Hess authored
      git-annex (5.20140717) unstable; urgency=high
      
        * Fix minor FD leak in journal code. Closes: #754608
        * direct: Fix handling of case where a work tree subdirectory cannot
          be written to due to permissions.
        * migrate: Avoid re-checksumming when migrating from hashE to hash backend.
        * uninit: Avoid failing final removal in some direct mode repositories
          due to file modes.
        * S3: Deal with AWS ACL configurations that do not allow creating or
          checking the location of a bucket, but only reading and writing content to
          it.
        * resolvemerge: New plumbing command that runs the automatic merge conflict
          resolver.
        * Deal with change in git 2.0 that made indirect mode merge conflict
          resolution leave behind old files.
        * sync: Fix git sync with local git remotes even when they don't have an
          annex.uuid set. (The assistant already did so.)
        * Set gcrypt-publish-participants when setting up a gcrypt repository,
          to avoid unncessary passphrase prompts.
          This is a security/usability tradeoff. To avoid exposing the gpg key
          ids who can decrypt the repository, users can unset
          gcrypt-publish-participants.
        * Install nautilus hooks even when ~/.local/share/nautilus/ does not yet
          exist, since it is not automatically created for Gnome 3 users.
        * Windows: Move .vbs files out of git\bin, to avoid that being in the
          PATH, which caused some weird breakage. (Thanks, divB)
        * Windows: Fix locking issue that prevented the webapp starting
          (since 5.20140707).
      
      # imported from the archive
      e213ef31
  30. Apr 17, 2014
    • Joey Hess's avatar
      replace (Key, Backend) with Key · e880d0d2
      Joey Hess authored
      Only fsck and reinject and the test suite used the Backend, and they can
      look it up as needed from the Key. This simplifies the code and also speeds
      it up.
      
      There is a small behavior change here. Before, all commands would warn when
      acting on an annexed file with an unknown backend. Now, only fsck and
      reinject show that warning.
      e880d0d2
  31. Apr 11, 2014
    • Joey Hess's avatar
      · 2a82adc4
      Joey Hess authored
      git-annex (5.20140412) unstable; urgency=high
      
        * Last release didn't quite fix the high cpu issue in all cases, this should.
      
      # imported from the archive
      2a82adc4
  32. Apr 02, 2014
    • Joey Hess's avatar
      · b6d46c21
      Joey Hess authored
      git-annex (5.20140402) unstable; urgency=medium
      
        * unannex, uninit: Avoid committing after every file is unannexed,
          for massive speedup.
        * --notify-finish switch will cause desktop notifications after each
          file upload/download/drop completes
          (using the dbus Desktop Notifications Specification)
        * --notify-start switch will show desktop notifications when each
          file upload/download starts.
        * webapp: Automatically install Nautilus integration scripts
          to get and drop files.
        * tahoe: Pass -d parameter before subcommand; putting it after
          the subcommand no longer works with tahoe-lafs version 1.10.
          (Thanks, Alberto Berti)
        * forget --drop-dead: Avoid removing the dead remote from the trust.log,
          so that if git remotes for it still exist anywhere, git annex info
          will still know it's dead and not show it.
        * git-annex-shell: Make configlist automatically initialize
          a remote git repository, as long as a git-annex branch has
          been pushed to it, to simplify setup of remote git repositories,
          including via gitolite.
        * add --include-dotfiles: New option, perhaps useful for backups.
        * Version 5.20140227 broke creation of glacier repositories,
          not including the datacenter and vault in their configuration.
          This bug is fixed, but glacier repositories set up with the broken
          version of git-annex need to have the datacenter and vault set
          in order to be usable. This can be done using git annex enableremote
          to add the missing settings. For details, see
          http://git-annex.branchable.com/bugs/problems_with_glacier/
        * Added required content configuration.
        * assistant: Improve ssh authorized keys line generated in local pairing
          or for a remote ssh server to set environment variables in an
          alternative way that works with the non-POSIX fish shell, as well
          as POSIX shells.
      
      # imported from the archive
      b6d46c21
  33. Mar 26, 2014
    • Joey Hess's avatar
      · f4a98d2d
      Joey Hess authored
      git-annex (5.20140320~bpo70+1) wheezy-backports; urgency=medium
      
        * Updating backport to newest release.
        * Note that this backport does not feature constant time webapp auth token
          comparisons. It's not recommended to use the webapp on multiuser systems,
          since another use may be able to use a timing attack to guess its auth
          token. If you need that, it should not be hard to backport
          haskell-securemem.
      
      # imported from the archive
      f4a98d2d
  34. Feb 20, 2014
    • Joey Hess's avatar
      · 9c426c0d
      Joey Hess authored
      git-annex (5.20140210~bpo70+2) wheezy-backports; urgency=medium
      
        * Updating backport to newest release.
        * Remaining differences in this backport:
          - No webdav special remote support.
          - Test suite is not built into git-annex as it now uses haskell-tasty,
            which is not yet backported.
          - No skein hash support.
      
      # imported from the archive
      9c426c0d
  35. Nov 27, 2013
    • Joey Hess's avatar
      · 7189dfd7
      Joey Hess authored
      git-annex (5.20131127) unstable; urgency=low
      
        * webapp: Detect when upgrades are available, and upgrade if the user
          desires.
          (Only when git-annex is installed using the prebuilt binaries
          from git-annex upstream, not from eg Debian.)
        * assistant: Detect when the git-annex binary is modified or replaced,
          and either prompt the user to restart the program, or automatically
          restart it.
        * annex.autoupgrade configures both the above upgrade behaviors.
        * Added support for quvi 0.9. Slightly suboptimal due to limitations in its
          interface compared with the old version.
        * Bug fix: annex.version did not get set on automatic upgrade to v5 direct
          mode repo, so the upgrade was performed repeatedly, slowing commands down.
        * webapp: Fix bug that broke switching between local repositories
          that use the new guarded direct mode.
        * Android: Fix stripping of the git-annex binary.
        * Android: Make terminal app show git-annex version number.
        * Android: Re-enable XMPP support.
        * reinject: Allow to be used in direct mode.
        * Futher improvements to git repo repair. Has now been tested in tens
          of thousands of intentionally damaged repos, and successfully
          repaired them all.
        * Allow use of --unused in bare repository.
      
      # imported from the archive
      7189dfd7
  36. Nov 06, 2013
    • Joey Hess's avatar
      · d2633388
      Joey Hess authored
      git-annex (4.20131106~bpo70+1) wheezy-backports; urgency=low
      
        * Backport is now built against git 1.8.4, also now available in backports.
        * Improve local pairing behavior when two computers both try to start
          the pairing process separately.
        * sync: Work even when the local git repository is new and empty,
          with no master branch.
        * gcrypt, bup: Fix bug that prevented using these special remotes
          with encryption=pubkey.
        * Fix enabling of gcrypt repository accessed over ssh;
          git-annex-shell gcryptsetup had a bug that caused it to fail
          with permission denied.
        * Fix zombie process that occurred when switching between repository
          views in the webapp.
        * map: Work when there are gcrypt remotes.
        * Fix build w/o webapp.
        * Fix exception handling bug that could cause .git/annex/index to be used
          for git commits outside the git-annex branch. Known to affect git-annex
          when used with the git shipped with Ubuntu 13.10.
      
      # imported from the archive
      d2633388