Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
wei_guo
Support
Support

Talend Studio : git project: 2GB hard-limit caused project login failure issue

Studio: git : 2GB hard-limit caused project login failure issue

Problem :
When customer's single project git repo exceeding 2G+ and studio gives Error in popup(see below),studio could not login projects.

Log :
An error occurred (object too large xxx bytes), rejecting the pack, Max object size limit is xxx bytes),

Analysis :
1. some big file has been committed mistakenly by developers they deleted and committed inside git, but the file is still persisted in git blobs cause this is git default mechanism to keep histroy operation with those big files;
2. current jgit version used by studio has hard-limit on total size of repo.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=531323
https://bugs.eclipse.org/bugs/show_bug.cgi?id=562838
(pls check if there is some big file has been committed to repo mistakenly)

How to fix (use BFG tool to identify those dangling big block file which has big file inside)?

1. BACKUP repo with fork operation from github/azure devops WebUI (in case of any git )
First clone a fresh copy of your repo, using the --mirror flag:
$ git clone --mirror git://example.com/some-big-repo.git
This is a bare repo, which means your normal files won't be visible, but it is a full copy of the Git database of your repository, and at this point you should make a backup of it to ensure you don't lose anything.
Now you can run the BFG to clean your repository up:
$ java -jar bfg.jar --strip-blobs-bigger-than 200M some-big-repo.git
The BFG will update your commits and all branches and tags so they are clean, but it doesn't physically delete the unwanted stuff. Examine the repo to make sure your history has been updated, and then use the standard git gc command to strip out the unwanted dirty data, which Git will now recognise as surplus to requirements:
$ cd some-big-repo.git
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Finally, once you're happy with the updated state of your repo, push it back up (note that because your clone command used the --mirror flag, this push will update all refs on your remote server):
$ git push

Ref: https://rtyley.github.io/bfg-repo-cleaner/ (edited)

Labels (3)
0 Replies